Image processing

ABSTRACT

Apparatus is provided to color-suppress a source frame that is composed of a plurality of pixels, each pixel being represented by three components defining a position within a color space. The source frame, comprising a foreground image against a background of a substantially uniform backing color, is provided to a processing means via an input means. A distance value is defined that for any pixel describes how close its color is to a specified color, and the processing means calculates a transformation that maximises the distance value of a selected background pixel. Then, for each pixel, the processing means calculates a transparency value that is a function of the distance value of the corresponding transformed pixel. The transparency value is used to calculate how much of the backing color should be subtracted. The foreground image can then be composited with a replacement background using the same transparency values.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to generating composited image datafrom respective background and foreground image data with an imageprocessing system configured with graphics data processing means.

DESCRIPTION OF THE RELATED ART

[0002] Recent technical advances in image processing systems havefacilitated and generalised the use of highly-saturated studioenvironments in which to film talents performing a scene, which shouldsubsequently be composited in an alternative background. “Bluescreen”techniques in video environments and “greenscreen” techniques incinematographic environments are well known, wherein talents or modelsare filmed in a studio configured with respectively blue- orgreen-saturated surroundings, in order to generate a clip of foregroundimage frames. An alternative clip of background image frames issubsequently generated and a compositing process allows an image editorto seamlessly blend the foreground and background image frames by meansof keying parts of the corresponding video signals, for instance theluminance or chrominance signal, which is known to those skilled in theart as chroma-keying.

[0003] Such image data processing has long been provided by means ofusing dedicated hardware, for instance the ultimatte provided by theUltimatte Corporation of Chatsworth, Calif. or, more recently, by meansof computer apparatus configured to process and output renderedcomposited image frames.

[0004] Image data processing apparatus as described above is veryexpensive, however, and traditionally requires similarly expensive,highly trained operators. Moreover, only a marginal portion of suchknown systems are configured with a processing capability sufficient togenerate composited image frames in real time, an accepted definition ofreal-time rendering being receiving, processing and outputting imagedata at display rate, also known to those skilled in the art as thenumber of displayable frames per second.

[0005] Techniques are known to overcome the elevated cost of ownershipdescribed above, which use relatively inexpensive computer systems asimage processing systems, and especially when configured with hardwarebased graphics accelerators, an example of which would be an NvidiaGEforce3 provided by the Nvidia Corporation of Santa Clara, Calif. Animportant problem however hinders the development of the use ofinexpensive computer systems as described above to generate compositedimage data, because hardware graphics accelerators are typicallydesigned to best process three-dimensional primitives such as polygonsbut the dedicated processing capabilities of the processors andsub-processors thereof are limited in relation to two-dimensional imageframe data processing algorithms, for instance the image keyingalgorithms developed by Petro Vlahos. According to the known prior art,such keying algorithms are computed by the main processor of an imageprocessing system, whereby all of the source, matte and destinationimage data is provided by said main processor to said graphicsaccelerator as textures. In the above described inexpensive computersystems, the processing of such source image data to generate said matteimage data and subsequent compositing thereof into said final image datarequires significant computation by said main processor, wherein thesupplementary graphical data processing capabilities of said graphicsaccelerator are not exploited which renders such inexpensive systemsunsuitable for outputting composited image data in real time. One reasonfor this is that graphics accelerators can usually only handlerelatively simple calculations while traditional compositing methodsrequire the solution of complex equations.

[0006] In Figure A an inexpensive computer system A1 is shown configuredwith a main processor A2, random-access memory A3 and data storage meansA4, wherein all of said components A2 to A4 are interfaced by means of asystem bus A5. A graphics accelerator A6 is also shown configured with amain graphic processor A7 and a plurality of graphics sub-processors A8,wherein processor A7 receives graphical data from processor A2 by meansof a dedicated bus A9 and forwards said graphical data after localprocessing to said sub-processors A8 for further processing. Saidprocessed graphical data is eventually written by said sub-processors A8to a frame buffer A10 from which said data is either output to a displayA11 in the form of image frames or sent back to main processor A2 overbus A9 for subsequent temporary storing in memory A3 or permanentstoring in storage means A4.

[0007] A functional overview of the components A7 to A10 of graphicsaccelerator A6 is shown in Figure B, in which a processing function ofgraphics processor A7 is to sort graphic data sent from processor A2between three-dimensional data and two-dimensional data. Typically, 3-Ddata is sent by processor A7 as control points equipped withco-ordinates in a volume configured with a Cartesian co-ordinate systemto a first evaluating system sub-processor A81, where vertices areevaluated from said control points. Said vertices are then sent fromsaid sub-processor A81 to a second vertex processing sub-processor A82,where they are converted into primitives, i.e. polygons. Two-dimensionaldata is sent by processor A7 to a third texture-processing sub-processorA83, a typical function of which is to generate positional data andcolor attributes, such that when the polygons are sent fromsub-processor A82 to a fourth rasterizing sub-processor A84 forconversion into screen-positioned, two-dimensional pixels, saidtwo-dimensional graphic data is similarly rasterized at a correctposition within the eventual image frame and, when said pixels are sentfrom said sub-processor A84 to a fifth fragment-processing sub-processorA85 for further color and/or transparency data processing, thetwo-dimensional graphic data-dependent color attributes are correctlyprocessed and associated with said pixels. In accordance with thedescription of Figure A, sub-processor A85 outputs final image framedata to frame buffer A10 but may optionally loop said output back tosaid texture-processing sub-processor A83 in order to reprocess saidoutput, for instance if data processing operations performed by saidsub-processor A85 require multiple passes.

[0008] In order to output image data, graphics accelerator A6 requiresinstructions and data from main processor A2, wherein said instructionsare processed by said processor A2 into microcode that graphicsprocessor A7 can then itself execute when received, and said microcodemay be received only once at start-up, for instance initialisationinstructions, or said instructions may be received on a regular basis toinstruct the accelerator 206 to perform certain tasks required by theapplication executed by said main processor A2. Examples of suchinstructions are shown in Figure C as OpenGL pseudo code, to improve theclarity of the present description.

[0009] Pseudo code C1 for instance sets a color and the example showninstructs graphics accelerator A6 to draw the brightest red possible,with no green or blue components. Pseudo code C2 similarly instructsgraphics accelerator A6 to draw a rectangle, and the joining of C1 andC2 would instruct accelerator A6 to draw an intensely red rectangle. Interms of invoking the specific functionality of sub-processors, pseudocode C3 instructs accelerator A6 to blend an intensely red rectanglewith an alternative background, whereby said rectangle data is processedby sub-processors A81 and A82, the color (red) data is processed bysub-processor A83 and the blending thereof is processed by fragmentprocessing sub-processor A85, with blending parameters specified at C4.

[0010] Further levels of sub-processor functionality and acceleratorconfiguration can be accessed at will in order to obtain the desiredoutput image data. For instance, complex pseudo code C5 initialises animage data processing mode known as “Feedback”, wherein no output isprovided to the displayable portion of frame buffer A10 until saidoutput data is fully processed and ready. Incoming graphics data is thuslooped through the various sub-processors, each of which carriesiterative functions thereon, until such time as all functions have beencarried out and the final data is now sent to said displayable portionof frame buffer A10. Such iterative function processing is traditionallyknown as rendering passes. Said sub-processors thus also requireinstructions shown as pseudo code C6 to C8 in order to determine whereto access (C6) said looped data and where to store (C7) or copy (C8)said data after they have processed it.

[0011] It is the pseudo code described in Figure C and executed by amain processor A2 that configures the graphics processor A7 and,further, the sub-processors A81 to A85 to perform their respective tasksdescribed in Figures B and C, wherein said code defines operationalsteps shown in Figure D for the purpose of said configuration.Typically, said code is compiled into a binary executable form which,when processed by said processor A2, is output as microcode, e.g.processing functions, to processor A7. Said microcode would thusinstruct graphics processor A7 to read incoming graphic data at step D1and discriminate said incoming data by asking a question at step D2 toestablish whether said incoming data is equipped with three-dimensionalattributes or not. If the question of step D2 is answered positively,said incoming data is sent to the texture processing sub-processor A83at step D3, where it is processed into a texture to be mapped ontopolygons. Alternatively, the question of D2 is answered negatively,whereby the three-dimensional data is sent to the combination ofsub-processors A81 and A82 at step D4, where it is processed intopolygons.

[0012] At step D5, both the three-dimensional data resulting from theexecution of step D4 and the two-dimensional data resulting from theexecution of step D3 are rasterized into two-dimensional screenco-ordinates by a process of rasterization, whereby the combination of2-D and 3-D data is known as fragments, i.e. all of the data associatedwith a pixel including coordinates, color, depth and textureco-ordinates. At this stage, fragments are traditionally temporarilywritten to the frame buffer A10 but at the next step D6, each of saidpixel/fragment is blended with the fragment color attributes such that adisplayable pixel is drawn back to the frame buffer at step D7. At thenext step D8, a question is asked as to whether the intended colorattributes of said displayable pixel require further processing beforesaid pixel is eventually output to display means and/or storage means.If the question of step D8 is answered positively, the first iterationof said fragments processed into displayable pixels according to step D6and written as a first iteration of said displayable pixels in saidframe buffer A10 according to step D7 are copied back to textureprocessing sub-processor A83, such that the next processing function,i.e. pass, may be performed thereon according to steps D5, D6 and D7.Alternatively, the question of step D8 is answered negatively, wherebyfinal processed graphic data is output to said display means and/orstorage means and control is returned to step D1 and next graphical datais read from main processor A2 by a graphical processor A7 and so on andso forth.

[0013] A three-dimensional cube is shown in Figure E within a volumeconfigured with a Cartesian co-ordinate system E1 defined by a set ofthree orthogonal axes. Said cube comprises six faces existing as square,textured polygons. With respect to the view adopted in Figure E, onlythree such textured polygons E2, E3 and E4 are visible. The cube shownin Figure E is a very simple three-dimensional structure representinggraphical data to be rendered and for which graphics accelerators suchas graphics accelerator A6 have been developed and optimised.

[0014] In accordance with the description of Figure A to D, eightcontrol points F1 to F8 are shown in Figure F, representingthree-dimensional data sent by graphics processor A7 to the evaluationsub-processor A81, wherein each of said control points is equipped withits respective three-dimensional, positional data in reference toco-ordinate system E1. Upon completing the evaluation processing of stepD4, sub-processor A81 eventually outputs twelve vertices F9 to F20, eachof which links two each of said 3-D control points F1 to F8 to defineone side of the cube object shown in Figure E

[0015] Upon completing the conversion processing at the same step D4,sub-processor A82 outputs six primitives G1 to G6 shown in Figure G,wherein said primitives are square, four-sided polygons defining thecube faces such as cube faces E2, E3 and E4. In the figure, polygon G2defines cube face E2, polygon G3 defines cube face E3 and polygon G4defines cube face E4 and polygons G1, G5 and G6 are occluded from view.The occlusion of said polygons, G1, G5 and G6 by said polygons G2, G3and G4 in relation to the view point with which the polygon object isshown in the figure results in only said visible polygons G2, G3 and G4being rasterized according to step D5, i.e. the pixels representing therespective surfaces thereof are processed in order to confertwo-dimensional screen co-ordinates thereto in relation to atwo-dimensional co-ordinate system G5 comprising two perpendicular axes.

[0016] In the art, the omission of the occluded polygons G1, G5 and G6from the rendering of the entire cube is known as culling and istraditionally performed by the fifth fragment processing sub-processorA85, typical operational steps of which are further described in FigureH.

[0017] It was previously explained that sub-processor A85 receives datacolloquially referred to as pixels, but which are better known in theart as fragments which are all of the data associated with any such onepixel, including two-dimensional co-ordinates, color, depth and textureco-ordinates. Typically, the fragment processor A85 blends thefragment's color and/or texture provided by sub-processor A83 with thepixel provided by sub-processor A84 and submits the fragments depth datato a culling test to determine whether said pixel is written to theframe buffer or not, e.g. a pixel generated from the service of polygonG1 would fail said depth-based test and thus not be rendered to theframe buffer. The above occluding function is one typical exampleamongst a plurality of functions carried out by sub-processor A85,whereby all of the fragments defining a complete image frame areiteratively processed by each function and, potentially, any of saidfunctions may thus require a subsequent reprocessing of the entire arrayof fragments, thereby defining multiple rendering passes.

[0018] At step H1, fragment processor A85 first reads the incomingfragment header, which specifies the fragment processing function withwhich the data therein should be processed. Said reading step prompts aquestion to be asked at the next step H2, to determine whether saidrequired function, which would usually be in the form of microcode asdescribed above, is already stored in a portion of dynamic memory ofgraphics accelerator A6 dedicated to storing processing functionsexecutable by said fragment processor A85. If the question of step H2 isanswered negatively, fragment processor A85 subsequently invokesgraphics processor A7 for the missing function, which is then fetchedand written for storage in said dedicated RAM portion at the next stepH4.

[0019] Alternatively, the question of step H2 is answered positively,signifying that said function already resides in said dedicated memoryportion, whereby fragment processor A85 may now process the fragmentread at step H1 with the appropriate function at the next step H5. Adisplayable pixel is thus obtained from said processing step H5 which issubsequently drawn to the frame buffer at step H6 but may equally bedrawn into a plurality of configurable portions of said frame bufferA10, known to those skilled in the art as color buffers, depth buffer,stencil buffer or an accumulation buffer, wherein said configurationsdepend upon the initialisation of graphics accelerator A6 by the imageprocessing application used and the parameterisation of the processingfunctions carried out therein.

[0020] At step H7, a question is asked as to whether the functionexecuted at step H5 specifies a reprocessing of the fragment read atstep H1, processed at step H5 and drawn as a pixel at step H6, i.e. asubsequent rendering pass is required before the required final outputpixel is displayed. If the question of step H7 is answered positively,the fragment processor A85 copies the buffered pixel of step H6 to thetexture processing sub-processor A83, whereby its two-dimensional screenco-ordinates will be sent to the rasterizing sub-processor A84 and itscolor and other such attributes will be sent back to fragment processorA85 in order to carry out said next processing function on what isessentially the output of the first iteration of the processing step H5.Alternatively, the question of step H7 is answered negatively, whereby athird question is asked at step H9, as to whether another fragment is tobe read and subsequently processed. Typically the question of step H9 isanswered positively as fragment processor A85 receives the fragmenthaving two-dimensional screen co-ordinates defining said fragment as thedisplayable pixel next to the last such processed displayable pixels inthe eventual output image frame from sub-processor A84, whereby controlis thus returned to step H1. Alternatively, the question of step H9 isanswered negatively, signifying that all of the displayable pixelsdefining a first iteration of the output image frame have beenprocessed, for instance to generate the first rendering pass asdescribed above, whereby the fragment processor memory is flushed at thenext step H10 and control is also returned to the initial reading stepH1, where for instance the first fragment processed by the firstfunction and copied to the texture processor A83 is read and the nextprocessing function corresponding to the above described secondrendering pass is carried out and so on and so forth, until all of thefragments defining the eventual output image frame have been processedby as many successive functions as required by fragment processor A85 togenerate the final output image frame.

[0021]FIG. 1 illustrates the rasterized polygon G3 shown in Figure Gconfigured with a texture as shown on cube face E3 in Figure E by meansof a processing function carried out by sub-processor A85, havingreceived the fragments defining said rasterized polygon G3 comprisingpositional data from rasterizer A84 and texture color attributes fromtexture processor A83. In accordance with the above description of theknown prior art, graphics processor A7 receives control points F1 to F8defining the cube object shown in Figure E, along with a texture I1which it determines to be two-dimensional data thus sent to textureprocessing sub-processor A83 shown as 12. Sub-processor A83 subsequentlyprocesses said texture I1 with scaling and biasing functions tocorrectly skew, position and orient said texture I1 in relation to theCartesian co-ordinates system E1 such that it overlays said polygon G3.Texture processor A83 thus generates a processed texture I3, havingpositional data derived from said processing functions shown in theFigure as I4 and color attributes I5, wherein said positional data issent to rasterizer A84 and said color attributes are sent to fragmentprocessor A85.

[0022] Fragment processor A85 thus receives all of the fragmentsdefining polygon G3 overlaid with texture I3 having color attributes I5within the two-dimensional orthogonal system G5, whereby said fragmentsare processed with a number of functions described above, an example ofwhich would be the culling function, configuring fragment processor A85to remove the portion of fragments defining polygons G1 and G6positioned behind polygon G3 in relation to the rendering view point.

BRIEF SUMMARY OF THE INVENTION

[0023] According to an aspect of the present invention, there isprovided image editing apparatus, comprising processing means, inputmeans and selection means; wherein a source frame comprising aforeground image against a background of a substantially uniform backingcolor is provided to said processing means via said input means, whereinsaid source frame is composed of a plurality of pixels, each pixel beingrepresented by three components defining a position within a colorspace; wherein said processing means is configured to define a distancevalue that, for any pixel in the source frame, describes how close itscolor is to a specified color; calculate a transformation of a selectedpixel in said background that maximises the distance value of saidselected background pixel; for each pixel in said source frame, define acorresponding transformed pixel by applying said transformation to saidsource frame pixel; for each pixel in said source frame, calculate atransparency value, wherein said transparency value is a function of thedistance value of the corresponding transformed pixel; and for eachpixel in said source frame, multiply the components of the selectedbackground pixel by the transparency value of said source frame pixeland subtract the result from the components of said source frame pixelto color-suppress said source frame.

[0024] According to a further aspect of the invention, there is provideda method of color-suppressing a source frame comprising a front imageagainst a background of a substantially uniform backing color, whereinsaid source frame is composed of a plurality of pixels and each pixel isrepresented by three components defining a position within a colorspace; comprising the steps of calculating a transformation thatmaximises a distance value of a selected pixel in the background, whichtransformation when applied to any pixel in said source frame defines acorresponding transformed pixel, wherein for any pixel said distancevalue describes how close its color is to a specified color; calculatinga transparency value for each pixel in the source frame, wherein foreach said pixel said transparency value is a function of the distancevalue of the corresponding transformed pixel; and, for each pixel in thesource frame, multiplying the components of said selected backgroundpixel by the transparency value of the source frame pixel andsubtracting the results from the components of the source frame pixel.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0025]FIG. 1 shows an image processing system operated by an artist,which comprises an inexpensive computer system;

[0026]FIG. 2 shows a typical internal architecture of the computersystem shown in FIG. 1, including a graphics accelerator card and amemory;

[0027]FIG. 3 shows a typical internal architecture of the graphicsaccelerator card shown in FIG. 2;

[0028]FIG. 4 details the operational steps according to which the artistshown in FIG. 1 operates the image processing system according to thepresent invention;

[0029]FIG. 5 details the contents of the memory shown in FIG. 2 uponcompleting the application starting step shown in FIG. 4, including animage processing application according to the present invention;

[0030]FIG. 6 further details the initialisation step shown in FIG. 4with which the application according to the present inventioninitialises the graphics accelerator card shown in FIGS. 2 and 3;

[0031]FIG. 7 provides a graphical representation of the graphical userinterface of the application displayed in FIG. 6, including a colorselection interface;

[0032]FIG. 8 further details the keying initialisation step shown inFIG. 4;

[0033]FIG. 9A shows image data captured by the camera shown in FIG. 1 assupplied to the computer system also shown in FIG. 1;

[0034]FIG. 9B details the captured image data shown in FIG. 9A,including a plurality of picture screen elements;

[0035]FIG. 9C shows values conferred to the picture screen elementsshown in FIG. 9B in order to generate a matte of the captured image datashown in FIGS. 9A and 9B;

[0036]FIG. 9D provides an alternative representation of the picturescreen elements defining a matte of the captured image data shown inFIG. 90;

[0037]FIG. 9E shows the matte shown in FIG. 9D configured by the presentinvention as foreground image data for subsequent compositing;

[0038]FIG. 9F shows captured image data shown in FIGS. 9A to 9Ecomposited with background image data according to the presentinvention;

[0039]FIG. 10 illustrates the compositing of a background image frameshown in FIG. 9F, the foreground image frame shown in FIG. 9A and thematte shown in FIG. 9E generated according to the present invention inthe frame buffer shown in FIGS. 3 and 6;

[0040]FIG. 11 shows a pixel's color values represented as a point in theRGB cube;

[0041]FIG. 12 illustrates an RGB cube viewed from the white corner;

[0042]FIG. 13 illustrates steps needed to rotate a point in RGB spacearound a line shown in FIG. 11;

[0043]FIG. 14 shows a matrix transformation that performs the stepsshown in FIG. 13;

[0044]FIG. 15 defines a notion of distance in the RGB cube;

[0045]FIG. 16 illustrates a method of determining an angle for rotationin the RGB cube;

[0046]FIG. 17 shows the RGB cube with a hue-compensated point;

[0047]FIG. 18 defines the notion of distance using hue-compensatedpoints;

[0048]FIGS. 19A and 19B illustrate planes formed in the RGB cube by thenotion of distance;

[0049]FIG. 20 details equations defining a transparency mask;

[0050]FIG. 21 illustrates planes in the RGB cube obtained duringcolor-suppression;

[0051]FIG. 22 illustrates a polygon defined by the planes shown in FIG.21;

[0052]FIG. 23 details calculations performed to obtain a transparencymask;

[0053]FIG. 24 shows calculations necessary to composite a foreground andbackground image;

[0054]FIG. 25 details calculations necessary for flare-suppression;

[0055]FIG. 26 details equations defining a flare-suppression mask;

[0056]FIG. 27 illustrates a plane obtained during flare-suppressionalong with the planes shown in FIG. 21;

[0057]FIG. 28 illustrates a polygon in RGB space obtained duringflare-suppression;

[0058]FIG. 29 details calculations performed to obtain aflare-suppression mask;

[0059]FIG. 30 details calculations carried out during flare-suppression;

[0060]FIG. 31 details calculations necessary to substitute the colorsuppression mask for the flare-suppression mask;

[0061]FIG. 32 details calculations necessary to perform colorsuppression;

[0062]FIG. 33 shows a graph illustrating flare-suppression and colorsuppression regions;

[0063]FIG. 34 shows equations illustrating the effect of colorsuppression and flare-suppression;

[0064]FIG. 35 shows the calculations necessary to perform colorsuppression and flare-suppression together;

[0065]FIG. 36 shows the equations necessary to perform onlyflare-suppression;

[0066]FIG. 37 shows the calculations necessary to perform only colorsuppression;

[0067]FIG. 38 details steps carried at step 406 as shown in FIG. 4wherein the user interacts with the application;

[0068]FIG. 39 shows a function carried out at step H5 in Figure H duringthe first pass of chroma-keying;

[0069]FIG. 40 shows a function carried out at step H5 in Figure H duringthe second pass of chroma-keying;

[0070]FIG. 41 shows a function carried out at step H5 in Figure H duringthe third pass of chroma-keying;

[0071]FIG. 42 shows a function carried out at step H5 in Figure H duringthe second pass of flare suppression only;

[0072]FIG. 43 shows a function carried out at step H5 in Figure H duringthe first pass of color suppression only;

[0073]FIG. 44 shows a function carried out at step H5 in Figure H duringthe second pass of color suppression only; and

[0074]FIG. 45 shows the final result of the analyst shown in FIG. 1composited with the background shown in FIG. 9F.

WRITTEN DESCRIPTION OF THE BEST MODE FOR CARRYING OUT THE INVENTION

[0075] The invention will now be described by way of example only withreference to the previously identified drawings.

[0076]FIG. 1

[0077] An image data processing system is shown in FIG. 1 and includes aprogrammable computer 101 having a drive 102 for receiving DVD-ROMs 103and writing to CD-RAMs 104 and a drive 105 for receiving high-capacitymagnetic disks, such as ZIP™ disks 106. Computer 101 may receive programinstructions via an appropriate DVD-ROM 103 and output data may bewritten to a re-writable CD-RAM 104. Program instructions may besimilarly received from a ZIP™ disk 106 and output data may be writtenthereto. Moreover, instructions may be transmitted to and received froma network server 107 or the internet 108, to which said server 107provides access, by means of network connection 109.

[0078] The user 110 of computer system 101 may visualise the output dataof computer 101 on a visual display unit 111. Manual input is receivedvia a keyboard 112, a mouse 113 and/or from a graphic tablet-and-styluscombination (not shown).

[0079] In the example, user 110 operates a video capture studio andsupplies image data for inclusion in webcasts, said webcasts beingbroadcasts streamed over the World Wide Web, or Internet. Accordingly,programmable computer 101 is interfaced with a camera 114, for instancea digital camera recorder supplying image data to said computer 101 bymeans of a Firewire™ connection 115. Alternatively, analogue image datamay be supplied from a conventional video camera and converted todigital data by an analogue to digital video capture interface. In theexample shown, user 110 has been tasked with supplying a commentary of astock market analyst 116 to a financial website in real time. Thus,analyst 116 provides her commentary, which is captured as digital videoby camera 114 in front of a bluescreen 117. Technical advances in imageprocessing systems have enabled the generalisation of the “bluescreen”technique in video, whereby actors or models are filmed in a studioconfigured with blue-saturated surroundings in order to generate a clipof foreground image frames. An alternative clip of background imageframes is subsequently generated and, in the example, said alternativeclip comprises various stock market graphical indicators. A compositingprocess allows an editor, such as user 110, to seamlessly blend theforeground and background image frames by means of keying part of thecorresponding video signals, for instance the luminance or chrominancesignal.

[0080]FIG. 2

[0081] The components of computer system 101 are further detailed inFIG. 2. The system includes a Pentium 4™ central processing unit (CPU)201 which fetches and executes instructions and manipulates data via aproviding connectivity with a larger main memory 203, DVD-ROM re-writer102, ZIP™ drive 105 and other components which will be further detailedbelow. System bus 202 is, for instance, a crossbar switch or other suchbus connectivity logic. CPU 201 is configured with a high-speed cache204 comprising between two hundred and fifty-six and five hundred andtwelve kilobytes, which stores frequently-accessed instructions and datato reduce fetching operations from larger memory 203. Memory 203comprises between two hundred and fifty-six megabytes and one gigabyteof dynamic randomly accessible memory and stores executable programswhich, along with data, are received via said bus 202 from a hard diskdrive 205. Hard disk drive (HDD) 205 provides non-volatile bulk storageof instructions and data.

[0082] A graphics card or accelerator 206 receives graphics data fromthe CPU 201, along with graphics instructions. Said graphics accelerator206 is preferably coupled to the CPU 201 by means of a direct port 207,such as the advanced graphics port (AGP) promulgated by IntelCorporation, the bandwidth of which exceeds the bandwidth of bus 202.Preferably, the graphics card 206 includes substantial dedicatedgraphical processing capabilities, so that the CPU 201 is not burdenedwith computationally intensive tasks for which it is not optimised.

[0083] Input/output interface 208 provides standard connectivity toperipherals such as keyboard 112, mouse 113, or a graphictablet-and-stylus. A Universal Serial Bus (USB) 209 is provided as analternative means of providing connectivity to peripherals such askeyboard 112, mouse 113 or said graphic tablet-and-stylus, whereby saidconnectivity is improved with a faster bandwidth for user input datatransfer. A Firewire™ interface 210 is also provided as an alternativemeans of providing connectivity to peripherals such as camera 114,whereby said connectivity is improved with a faster bandwidth for imagedata transfer.

[0084] Network card 211 provides connectivity to server 107 and theInternet 108 by processing incoming and outgoing data packets accordingto a plurality of communication protocols. Optionally, a sound card 212is provided which receives sound data from the CPU 201 over system bus202, along with sound processing instructions, in a manner similar tographics card 206. Preferably, the sound card 212 includes substantialdedicated digital sound processing capabilities, so that the CPU 201 isnot burdened with computationally intensive tasks for which it is notoptimised.

[0085] The equipment shown in FIG. 2 constitutes an inexpensiveprogrammable computer of fairly standard type, such as a programmablecomputer known to those skilled in the art as an IBM™ PC compatible oran Apple™ Mac.

[0086]FIG. 3

[0087] Instructions according to the present invention may be processedby the image processing system 101 to display 3-D graphical objects onthe video display unit 111, wherein the CPU 201 may transfer informationto and from the 3-D graphics accelerator 206 according to a programmedinput/output protocol over the AGP port 207. The 3-D graphicsaccelerator 206 in computer system 101 provides increased performancefor processing geometric primitives to be rendered as three-dimensionalobjects on VDU 111, thus reducing the need for an expensive workstationand the associated cost thereof.

[0088] Functionally, instructions according to the invention preferablyconform to an application programmer interface (API) such as OpenGLwhich, when processed by CPU 201, generate processor commands known asmicrocode and data that define a geometric primitive, such as a polygon,for output on VDU 111. The OpenGL (Open Graphics Library) API used inthe preferred embodiment is designed as a hardware-independent interfaceto be implemented on many different hardware platforms, but it will bereadily apparent to those skilled in the art that hardware-specific APIsor operating system-specific APIs may be used to achieve the benefit ofthe present invention. Such hardware-specific APIs or operatingsystem-specific APIs include for instance Nvidia's Cg™ ProgrammingLanguage, Creative Labs™'s Shading Programming Language, Microsoft™'sDirectX™ Pixel Shaders or Renderman™'s Image Shaders.

[0089] A typical architecture of a 3-D graphics accelerator 206 is shownin further detail in FIG. 3. The 3-D graphics accelerator 206principally comprises a graphics processing unit 301 (GPU) configuredwith a memory cache 302 and a dynamic memory 303. The accelerator 206further comprises a plurality of sub-processors including floatingpoints processors and drawing controllers as well as a frame buffer anda random access memory digital-to-analogue converter (RAMDAC). The GPU301 interfaces the graphics accelerator 206 to the AGP BUS 207 andcontrols the transfer of graphics processing instructions, known asmicrocode, and graphical data between said dynamic memory, floatingpoint processors, drawing controllers and frame buffer.

[0090] A first floating point processor 304 acts as an evaluator, theprocessing function of which derives the vertices used to represent apolygonal surface from its control points. A second floating pointprocessor 305 converts the vertices output by evaluator 304 intoprimitives, whereby the spatial co-ordinates thereof are projected froma position within a three-dimensional co-ordinate system to atwo-dimensional position on a screen. An additional function of vertexprocessor 305 is the assembly of said primitives, which involvesclipping (the elimination of geometrical data not required for output),viewport and depth processing operations and optionally culling, whereinall such processing operations will be familiar to those skilled in theart. The output of vertex processor 305 is complete geometricprimitives, including vertices with related color data, depth data,optionally texture coordinate values and format for the nextrasterization step by the rasterizer sub-processor 306, which will befurther described below. A first drawing controller 307 processesnon-geometric data received by GPU 301. Pixel processor 307 typicallyscales, biases and processes said non-geometrical data by means of apixel map, whereby its output is then either forwarded to a seconddrawing controller 308, the function of which is to apply texture imagedata onto geometric objects, for instance to make said objects look morerealistic, or said output is directly forwarded to said rasterizer 306.The rasterizer sub-processor 306 processes both geometric output data ofvertex processor 305 and pixel output data of pixel processor 307 and/ortexture assembler 308 into fragments, wherein each of said fragmentscorresponds to a pixel to be stored in frame buffer 309, which comprisesan array of addressable dynamic random access memory.

[0091] A third drawing controller 310 performs a series of operationsthat may alter or possibly eliminate fragments before they are stored inframe buffer 309. Fragment processor 310 primarily processes texture(s)received from texture assembler 308 to generate a texture element, alsoknown as a texel, to be applied to a fragment. Fragment processor 310may perform additional processing functions including fog calculations,pixel blending, pixel dithering and/or bitmask masking. The processedfragment is eventually drawn into the frame buffer 309, wherein it isnow a fully-processed, displayable pixel.

[0092] The RAMDAC 311 comprises a programmable video timing generatorand programmable pixel clock synthesiser along with crossbar functions,as well as traditional color look-up tables and triple video DACcircuits. RAMDAC 311 in turn couples to the video display unit 111. Thearchitecture of the 3-D graphics accelerator 205 described above mayvary to a large extent and is here only provided for illustrativepurposes. Those skilled in the art will be familiar with thefunctionality and performance benefits thereof.

[0093] With reference to the description of FIGS. 1 to 3, an aim of thepresent invention is to facilitate the generation of a matte from animage frame to composite a portion of said image, defined as aforeground image, with a background image from an alternative source.According to the known prior art however, the image processing system101 described in FIGS. 1 to 3 may not provide sufficient processingresources to generate a matte of the above described blue-screened imageframe in real-time as required. Indeed, although matte generation is awell-known technique for compositing image data from various sources, itis traditionally performed by the main processor 201 of the imageprocessing system 101, because the computations involved according tothe known prior art far exceed the processing capacities of any of thesub-processors in the graphics accelerator 206, and GPU 301 alone maynot perform those processes in real-time. Thus it is known to generate amatte from image data with CPU 201, which is then sent to GPU 301 as atexture. According to the preferred embodiment of the present invention,however, only the “bluescreen” image frame captured by camera 114 issent to GPU 301, and the matte thereof is generated by graphicsaccelerator 206.

[0094]FIG. 4

[0095] The operational steps according to which artist 110 operatesimage processing system 101 are detailed in FIG. 4.

[0096] At step 401, the computer system 101 is switched on, whereby allinstructions and data sets necessary to process image data are loaded atstep 402, including instructions according to the present invention togenerate a matte of said image data. Optionally, said instructions anddata sets necessary to process image data are loaded from a DVD-ROM 103,from network server 107 or the internet 108 at step 402, for instance ifsaid instructions are not yet stored on HDD 205. Upon completing theloading operation of step 402, the processing of said instructionsaccording to the present invention by CPU 201 starts at step 403.

[0097] A portion of the output of said processing step 403 comprisesmicrocode for the initialisation of graphics accelerator 206 at the nextstep 404, whereby CPU 201 sends said microcode to GPU 301. Saidmicrocode includes for instance parameterization for the operating modeand image data output of accelerator 206 and, preferably, processingfunctions to be performed by GPU 301 and its sub-processors 304 to 308and 310, which are stored in memory 303 and/or cache 302. The completionof the above initialisation step 404 results in the eventual output ofthe Graphical User Interface (GUI) of the image processing applicationaccording to the present invention by graphics accelerator 206 to VDU111, whereby user 110 may then select a preferred image keying processat step 405 from difference-keying, luminance-keying and chroma-keying,each of which will be further described below.

[0098] At step 406, image data is acquired as a single frame or,alternatively, from a clip of frames or stream thereof either inreal-time from camera 114 or the Internet 108 or from hard disk drive205, network server 107, a DVD-ROM 104 or a disk 106 such that it can bedisplayed to user 110 on VDU 111 for subsequent interaction therewith atstep 406 by means of the image processing application. In the preferredembodiment of the present invention, said image data is acquired inreal-time as a stream of image frames from camera 114. Upon completingthe interactive step 406, the user-inputted parameters thereof specifyhow to process image data in order to generate a matte according to thepresent invention at step 407.

[0099] At step 408, a question is asked as to whether another imageframe or another clip of image frames, i.e. a subsequent stream of imageframes, require processing by image processing system 101 according tothe present invention. If the question of step 408 is answeredpositively, control is returned to step 406 such that new image data canbe acquired in real-time from camera 114 or the internet 108, or fromhard disk drive 205, network server 107, a DVD-ROM 104 or a disk 106.Alternatively, if the question asked at step 408 is answered negatively,then user 110 is at liberty to stop the processing of the instructionsaccording to the present invention at step 409 and, eventually, switchimage processing system 101 off at step 410.

[0100]FIG. 5

[0101] The contents of main memory 203 subsequent to the applicationprocessing start step 403 are further detailed in FIG. 5. An operatingsystem is shown at 501 which comprises a reduced set of instructions forCPU 201, the purpose of which is to provide image processing system 101with basic functionality. Examples of basic functions include forinstance access to files stored on hard disk drive 205 or accessed fromDVD/CD ROM drive 102 or ZIP drive 105 and management thereof, networkconnectivity with network server 107, the Internet 108 and camera 114,interpretation and processing of the input from keyboard 112, mouse 113or graphic tablet-and-stylus. In the example, the operating system isWindows 2000 Professional™ provided by the Microsoft Corporation ofRedmond, Calif., but it will be apparent to those skilled in the artthat the instructions according to the present invention may be easilyadapted to function under different other known operating systems, suchas IRIX™ provided by Silicon Graphics Inc, Mac OGX or LINUX, which isfreely distributed.

[0102] An internet browser application is shown at 502, which processeshypertext markup language-coded data packets or other internet-specificformatted data or applets into graphical data for display on VDU 111. Adigital video capture application is shown at 503 which processesdigital video image data generated by camera 114 and accessed atFirewire™ interface 210 into graphical data for display on VDU 111 and,optionally, storage in HDD 205 CD RAM 104, server 107 or disk 106.

[0103] An image processing application according to the presentinvention is shown at 504, which configures graphics accelerator 206 togenerate a matte of the image data output by application 503 forsubsequent compositing. Corresponding application data is shown at 505which comprises various sets of user-independent data and userdependent-data according to which application 504 processes image data.Image data generated by application 503 to be subsequently processed byapplication 504 and its corresponding data 505 is shown at 506 andgeometric data generated by any of applications 501 to 504 forsubsequent processing by graphics accelerator 206 is shown at 507.Finally, user input data is shown at 508 which comprises userinput-dependent data identifying parameters and/or data input by user110 by means of keyboard 112, mouse 113 and/or graphic tablet-and-stylusto process image data 506.

[0104] Various datasets of each subset of main memory 203 are suppliedto GPU 301 for dedicated graphics processing by CPU 201. For instance,operating system 501 supplies device drivers 509 when image processingsystem 101 is initially switched on to ensure hardware compatibility.Similarly, image processing application 504 and application data 505supply microcode 510, wherein application data 505 specifies processingparameters within said microcode according to user input data 508. Imagedata 506 may be supplied to GPU 301 as bitmaps 511 or complete pixels512 either by application 503 or by application 504 and geometrical data507 may similarly be supplied to GPU 301 as lines or vectors 513,vertices 514 or complete polygons 515 either by application 503 or byapplication 504.

[0105] For the purpose of clarity, the specific description willhereinafter refer only to image processing application 504 supplyingmicrocode 510, pixels 512 defining a stream of image frames captured bycamera 114 and vertices 514.

[0106]FIG. 6

[0107] Before image processing application 504 may supply any graphicaldata, either geometric or non-geometric, to GPU 301, the graphicsaccelerator 206 must first be initialised by said application 504according to step 404, which is further detailed in FIG. 6.

[0108] Upon user 110 starting application 504 at step 403 CPU 201 parsesand processes the application's initial settings at step 601. Saidsettings for instance specify the screen resolution at which application504 operates, whereby the operating system 501 default displayresolution may be 1024×768 pixels but application 504 requires a displayresolution of 1600×1200 pixels, i.e. to display image data in finerdetail. Other such initial settings may include an API operating mode,indicating for instance whether graphics accelerator 206 should operateunder OpenGL, DirectX's Direct3D or other mode. CPU 201 subsequentlyforwards the corresponding initialising microcode to GPU 301, whereby itis initialised at step 602. Having processed said initialisingmicrocode, GPU 301 is thus able to initialise all of its sub-processors302 to 311 in turn at step 603.

[0109] At the next step 604, image processing application 504 anddefault application data 505 is generated by CPU 201 asfunction-specific microcode 510 and forwarded to GPU 301, whereby saidmicrocode 510 is subsequently stored by GPU 301 either in cache 302,dynamic memory 304, or a combination thereof at step 605. A question issubsequently asked at 606 as to whether a portion of saidfunction-specific microcode 510 instructs GPU 301 to configure a portionof frame buffer 309 as a temporary buffer, within which successiveiterations of fragments will be stored during processing. If thequestion of step 606 is answered positively, GPU 301 apportions saidaddressable dynamic RAM 309 according to the needs of application 504,specified in said microcode 510. The combination of GPU 301 and itssub-processors is thus fully initialised for generating the output dataof application 504, whereby the default Graphical User Interface (GUI)may now be output to VDU 111 at step 608. Alternatively, the question ofstep 606 is answered negatively, such that there is no requirement for atemporary buffer, thus control is directly forwarded to said step 608,whereby the default Graphical User Interface (GUI) may now be output toVDU 111 at step 608.

[0110]FIG. 7

[0111] A graphical representation of the GUI of application 504generated according to step 608 is shown in FIG. 7, including a colorselection interface.

[0112] VDU 111 is shown and displays the GUI 701 of operating system501. Said GUI 701 includes at least one user-actionable widget 702,presenting user 110 with standard operating system tasks such as filebrowsing upon activation. The GUI 703 of image processing application504 is shown overlaid over GUI 701 and includes a plurality ofuser-operable widgets, the functionality of some of which specifyparameters, i.e. application data 505.

[0113] GUI 703 includes a first display area 704 within which image dataacquired at step 406 is displayed to user 110 for interaction therewith.In the example, first image data comprises analyst 116 standing in frontof bluescreen 117, as captured by digital camera 114 and initiallyprocessed by digital video capture application 503. Within said displayarea 704, a moveable color selection interface 705 is provided for user110 to intuitively select the red, green and blue color component valuesof a pixel defining the background color to be keyed out. Preferably,the two-dimensional coordinates of interface 705 within area 704 areupdated in real-time from the two-dimensional co-ordinates defined bythe translation of mouse 113 or a stylus over a graphic tablet.

[0114] Upon positioning interface 705 satisfactorily, user 110 maysubsequently either click on a pressure-sensitive button of mouse 113 orpress a key of keyboard 112 to activate his selection, wherein the RGBcolor component values of the pixel currently designated by interface705 are read as a first parameter and stored in application data 505.GUI 703 includes a second display area 706 configured with user-operablesliders, the function of which is to finely adjust the values derivedfrom the above selection with interface 705. Thus area 706 is configuredwith sliders 707, 708 and 709 ranging between θ (no color) to 255 (fullcolor) for each of the red, green and blue color componentsrespectively.

[0115] A third display area 710 is provided within which further slidersmay be operated in a manner similar to sliders 707, 708 and 709 in orderto provide image processing application 504 with additional applicationdata 505. Within said area 710, a first slider 711 is provided to allowuser 110 to specify how much application 504 is tolerant to divergencebetween the foreground colors and the background colors. A second slider712 is provided to allow user 110 to configure how much softnessapplication 504 should apply to edges within image 704. Finally a thirdslider 713 is provided to allow user 110 to refine the level of softnessapplied by application 504, which is known according to the presentinvention as flare suppression. Each of sliders 711 to 713 preferablyranges between θ and 1, whereby the value identified by the position ofthe selector in said sliders is a floating point value.

[0116]FIG. 8

[0117] With the GUI 703 displayed onto VDU 111, user 110 may now selecta preferred image keying process according to step 405, which isdescribed in further detail in FIG. 8.

[0118] In the preferred embodiment, user 110 may select one of threeavailable image keying processing functions f(DK), f(LK) and f(CK) bymeans of activating keys of keyboard 112 mapped to each of said imagekeying processing functions. Alternatively, GUI 703 may provide aselection interface for the same purpose, for instance by means of a“keying selection” dropdown menu or even by means of a user-operablepop-up menu, both types of which are known. According to the presentembodiment, all three functions f(DK), f(LK) and f(CK) are stored asmicrocode in memory 303 and/or cache 302 upon completing theinitialisation step 404.

[0119] At step 801, a first question is asked as to whether user inputhas been received to select difference keying as the appropriate imagekeying processing function, f(DK). If the question of step 801 isanswered positively, said user input data is sent by CPU 201 afterinterpretation to GPU 301, which in turn configures fragment processor310 according to the present invention to fetch then process said imagekeying processing function f(DK) at step 802.

[0120] Difference keying involves recovering the transparency mask ofanalyst 116 within a foreground from an identical reference backgroundwhich must be provided, whereby said background-without-analyst andforeground-with-analyst are compared for “difference”, said differencebeing said transparency mask. User 110 may alter parameters in thefunction such as the threshold τ controlling how much the function istolerant to divergence of the foreground and background colors, rangingfrom θ to 1, and the softness σ controlling the softness of the edges inthe image to be matted, also ranging from θ to 1.

[0121] The keying function f(DK) is implemented in microcode andprocessed by fragment processor 310 as an equation:$a = {{clamp}( {\frac{{{C - C_{b}}}^{2} - \tau}{\Delta},0,1} )}$

[0122] where C is the input color pixel, C_(b) is the referencebackground color pixel and Δ=3[1−r−(1−σ)(1−τ)]. After a is computed asabove, the color value of each pixel is processed with the followingequation to remove the color of the background:

C _(f) =C−(1−a)C _(b)

[0123] The final pixel C_(f) contains the foreground image, i.e. analyst116, along with the backing color removed by means of its opacity valuea.

[0124] Alternatively, the first question of step 801 is answerednegatively, whereby a second question is asked at step 803 as to whetheruser input has been received to select luminance keying as theappropriate image keying processing function, f(LK). If the question ofstep 803 is answered positively, said user input data is sent by CPU 201after interpretation to GPU 301, which in turn configures fragmentprocessor 310 according to the present invention to fetch then processsaid image keying processing function f(LK) at step 804.

[0125] Luminance keying involves extracting a matte of an image framehaving image data which matches a user-defined luminance value, theadvantage of this processing function being that a specific luminancerange may be specified independently of the hue data. The keyingfunction f(LK) is implemented in microcode and processed by fragmentprocessor 310 as an equation:$a = {{clamp}( {( \frac{{C} - \tau}{\Delta} )^{2},0,1} )}$

[0126] where C is the input color pixel, τ is a threshold value and Δ isa scaling value.

[0127] Alternatively, the second question of step 803 is answerednegatively, whereby a third and final question is asked at step 805 asto whether user input has been received to select chroma-keying as theappropriate image keying processing function, f(CK). If the question ofstep 805 is answered positively, said user input data is sent by CPU 201after interpretation to GPU 301, which in turn configures fragmentprocessor 310 according to the present invention to fetch then processsaid image keying processing function f(CK) at step 806. Alternatively,the third question of step 805 is answered negatively, whereby controlis returned to question 801 and the process steps are repeated asdescribed above until such time as an image keying processing functionis selected.

[0128] The present description of the preferred embodiment willhereinafter describe the chroma-keying function f(CK) processed byfragment processor 310 in further detail, but it will be understood bythose skilled in the art that the present embodiment is not limited tosaid function nor is it limited within the context of a graphicsaccelerator to the use of said fragment processor.

[0129]FIG. 9A

[0130]FIG. 9A shows a first source frame 901, ie image data acquired atstep 406, depicting a foreground image of analyst 116 over asubstantially uniform background of a blue backing color 117, of astream of such image frames captured by the camera 114, wherein thefrequency of said stream is defined as the shutter speed of said camera,which in the example is twenty-five frames per second corresponding tothe European PAL broadcast format. Thus, image capture application 503preferably processes twenty-five image frames with sound per second intoimage data with a format acceptable by image processing application 504for subsequent further processing.

[0131]FIG. 9B

[0132]FIG. 9B shows said first source frame 901 after first processingby said image capture application 503, wherein said source frame is nowconfigured into a finite number of picture screen elements (pixels)defining the total size of the frame, i.e. its resolution, which dependsupon the resolution capability of the camera 114 itself, the formatselected for image capture application 503 output image data, forinstance Bitmap, JPEG or an alternative such image data format, whethercompressed or uncompressed.

[0133] The image data is thus shown over which a mesh 911 has beensuperimposed to figuratively represent the resolution thereof in pixels.It will be apparent to those skilled in the art that said pixels areshown artificially enlarged for clarity and illustrative purposes only.In effect, first display area 704 of the image processing application504 according to the invention displays pixels 911 according to theimage data acquisition step 406, before the matte thereof is generatedaccording to the user parameter selection step 406. Pixels 911 are thussupplied to GPU 301 under the form of four vertices, respectively theupper right corner, upper left corner, lower right corner and lower leftcorner of the array of pixels 911, i.e. the image frame, and a texturecomprising all of the pixels 911 within the array, including their colorcharacteristics.

[0134]FIG. 9C

[0135]FIG. 9C shows values conferred to the picture screen elementsshown 911 in FIG. 9B in order to generate a matte of the captured imagedata 901.

[0136] The RGB color component values of the pixels representing theblue background 117 have a relatively similar configuration, which isthe background's blue property, whilst the RGB color component values ofthe pixels representing the analyst 116 have a dissimilar configuration,including variations based upon skin tone, hair color, garment color andany other image component having RGB color component values differingsubstantially from those defining the uniform blue background.

[0137] For the purpose of clarity, pixels 911 are conferred values ofeither θ or 1, representing a condition of “false” or “true”respectively, in the description of this figure only, whereby a moredetailed description of said pixel evaluation will be providedhereinafter. According to the present invention, as the pixels definingthe blue background 117 have fairly uniform RGB values selected withcolor selection interface 705, said pixels are processed by fragmentprocessor 310 as defining the background to be removed and thusconferred a value 921 of θ. Conversely, the pixels defining the analyst116 have dissimilar RGB values and are thus processed by fragmentprocessor 310 and conferred a value 922 of 1, indicating they are to beretained.

[0138]FIG. 9D

[0139]FIG. 9D provides an alternative representation of the conditionalvalues 921, 922 applied to the pixels 911 shown in FIG. 9C, whereinpixels defining the uniform blue background 117 having a value of 0 arewhited out and pixels defining analyst 116 having a value of 1 areblacked out. It can therefore be observed that all pixels 911, 922 inthe image frame 901 now have a uniform property, thereby defining a mask931.

[0140]FIG. 9E

[0141]FIG. 9E shows the matte shown in FIG. 9D configured asalpha-channel image data according to the present invention forsubsequent compositing.

[0142] In order to successfully blend the pixels defining the backgroundimage data from an alternative source, fragment processor 310 processesthe first iteration of the rasterized source frame including the analyst116 and the blue background 117 according to the present invention. Theexample mask 931 thus derived from said rasterized source frame by saidfragment processor 310 is an image texture 941 stored in the temporarybuffer portion of frame buffer 309. Fragment processor 310 processes therespective red, green, blue and alpha color component values of thepixels defining analyst 116 and uniform blue background 117 with thechroma-keying function f(CK), whereby said values remain unchangedinsofar as analyst 116 is concerned but are nulled, save for a maximumtransparency alpha-channel value with regard to blue background 117.

[0143]FIG. 9F

[0144]FIG. 9F shows captured image data shown in FIGS. 9A to 9Dcomposited with background image data.

[0145] Further processing operations are performed by fragment processor310, notably an alpha-channel blending operation known as “alphablending”, comprising blending the pixels 911 shown in FIG. 9B, thepixels defining alternative background image data 951 and the texture941, whereby analyst 116 of texture 941 is correctly composited and onlythe black, conditionally “false” buffered pixel values are blended withsaid alternative pixels 951. Said blending operation is well-known tothose skilled in the art and is for instance described in the “OpenGLProgramming Guide”, Third Edition, Addison-Wesley, ISBN 0201604582, atpp.220-232.

[0146]FIG. 10

[0147] The processing according to the present invention of thebackground frame 951 shown in FIG. 9F, the source frame 901 shown inFIG. 9A and the alpha-channel frame 941 shown in FIG. 9E is furtherillustrated in FIG. 10, in relation to the interactions between theframe buffer 309 and the fragment processor 310.

[0148] In the preferred embodiment of the present invention, the framebuffer 309 of graphics accelerator 206 is configured with a temporarybuffer portion 1001 and a displayable image frame buffer portion 1002,upon GPU 301 receiving initialisation microcode 510 at step 404, wherebythe temporary buffer 1001 is specified according to step 607. In theexample, user 110 selects chroma-keying at step 405 by answeringquestion 805 positively, whereby the chroma-keying function f(CK)portion of microcode 510 will be invoked and processed by fragmentprocessor 310, to the exclusion of difference keying function f(DK) andluminance keying function f(LK).

[0149] First source frame 901 is received by pixel processor 307 at1003, whereby the color attributes thereof are sent to fragmentprocessor 310 at 1004, such that a first array of displayable pixels1005 is generated therein and stored in portion 1001, having beenprocessed as fragments including the rectangular shape of frame 901 as athree-dimensional data rasterized by rasterizer 306 and the imagecomponents, e.g. analyst 116 and blue background 117, as its colorattributes. In accordance with the present description, the contents oftemporary buffer 1001 are subsequently looped back to pixel processor307 and thus a fragment processor 310 at 1006, whereby said fragmentprocessor 310 generates the matte 941 of frame 901 and thus draws acorresponding second array of pixels 1007 in temporary buffer 1001,which may be understood as the alpha channel-based array of pixels ofarray 1005. The replacement background image frame data 951 is thensupplied to pixel processor 307 at 1008 whereby, in a manner similar tosource frame data 901, it is sent to fragment processor 310 and a thirdarray of pixels 1010 thereof is drawn into temporary buffer 1001. As wasthe case for the data path 1006, the contents 1005, 1007 and 1010 oftemporary buffer 1001 are looped back at 1111 to pixel processor 307 andthus eventually to fragment processor 310, whereby a subsequent blendingfunction is processed to composite said contents into a final outputimage frame 1112.

[0150] Thus, according to the present invention, the extraction of thematte 941 from image data 901 is carried out by fragment processor 310within graphics accelerator 206 and chroma-keying is performed withinsaid accelerator 206, thereby configuring inexpensive image processingsystem 101 with a real-time image compositing processing capabilitytraditionally reserved to much more expensive, specialist apparatus. Toachieve this benefit, the chroma-keying function f(CK) is itselfconfigured to process the red, green, blue and alpha values of eachpixel as a low-level mathematical operation that may be processed by aslimited a processor as fragment processor 310, an example of which isdescribed further below.

[0151]FIG. 11

[0152] There are many ways of representing specific colors ascoordinates in three-dimensional space. The RGB system is based on howmuch of each of the three primary colors red, green and blue is neededto produce a specified color. These values are then considered asco-ordinates in the RGB cube. Typically, values range either from zeroto one or from zero to two hundred and fifty-five. The followingdescription is based on values in the [0, 1] range and if values in the[0, 255] are used then they can be normalised by dividing through by twohundred and fifty-five.

[0153]FIG. 11 shows RGB cube 1101. Three-dimensional space is defined byred R-axis 1102, green G-axis 1103 and blue B-axis 1104. The cube isdefined by all points having values in the range [0, 1] on all threeaxes. Black, which is the absence of color, is at the origin, whilewhite, which is the combination of the maximum of all three colors, isat the point (1, 1, 1). The primary colors red, green and blue are atpoints (1, 0, 0), (0, 1, 0) and (0, 0, 1) respectively, while thesecondary colors yellow, magenta and cyan are at points (1, 1, 0), (1,0, 1) and (0, 1, 1) respectively.

[0154] Shades of grey are achieved when the co-ordinates on all threeaxes are equal, and thus all greys lie on line 1105, defined as R=G=B,which runs from the black corner to the white corner.

[0155] Point 1106 represents the color of the blue screen behind theanalyst in FIG. 1. It lies at the co-ordinates (0.2, 0.3, 0.8).

[0156] Other color systems use the concept of hue, saturation andluminance. For a given color, its luminance value is a linear functionof its R, G and B values. Thus, black has the least luminance whilewhite and the primary and secondary colors have maximum luminance.Roughly, therefore, increasing a color's luminance in RGB space meansmoving it away from black and towards white, in other words in thedirection shown by line 1105.

[0157] Saturation is a concept defined by how far a particular color isfrom the shade of grey with equal luminance. The most saturated colorsare the primary and secondary colors at the corners of the cube, whileblack, white and the greys in-between have no saturation. For example,point 1106 has a saturation value of 75%. Roughly, therefore, toincrease the saturation of a specific color within the RGB cube it mustbe moved towards the edges of the cube and away from line 1105.

[0158]FIG. 12

[0159]FIG. 12 shows the RGB cube viewed from the white corner, i.e. downline 1105. The concept of hue is defined by the rotational position of acolor when the cube is viewed in this way. It is traditionally measuredfrom red, so that red is at zero degrees, yellow at sixty degrees, greenat one hundred and twenty degrees, cyan at one hundred and eightydegrees, blue at two hundred and forty degrees and magenta at threehundred degrees.

[0160] It will now be clear that rotating a point about line 1105changes neither the luminance nor the saturation of the color but onlychanges the hue. This is known as a hue shift. For example, point 1106shown in FIG. 11 is also shown in FIG. 12. In this view of the RGB cubeit is easy to see that the hue of this color lies between blue and cyanand that the color is mediumly saturated. The luminance cannot be shownsince line 1105 extends out of the page. Rotating point 1106 by theangle shown at 1201 moves it to point 1202, which has the samesaturation and luminance as point 1106 but it is now the bluest that itcan be without changing these values. Perceptually, the color is neitherbrighter nor more vivid, but simply a different hue.

[0161]FIG. 13

[0162]FIG. 13 illustrates the steps needed to rotate a point in RGBspace around line 1105, ie hue shift the color, by an angle γ. At step1301 the point is rotated by 45° about the red axis and at step 1302 thepoint is rotated by 35.3° about the green axis. Twice-rotating line 1105itself in this way translates it onto the blue axis and so at step 1303the point is rotated by the desired angle γ around the blue axis. Thepoint is then rotated by −35.3° about the green axis at step 1304 and by−45° about the red axis at step 1305 to return it to the RGBcolor-space.

[0163]FIG. 14

[0164] Rotations around the red, green and blue axes are well definedand combining these five rotations results in matrix 1401, called matrixT, as shown in FIG. 14. The transformation of a pixel C to itscorresponding hue-shifted (transformed) pixel C is therefore defined asC=TC.

[0165]FIG. 15

[0166] This idea of changing a color's hue but not its saturation orluminance can be used in chroma-keying. If the talent is filmed againsta background of a uniform backing color, a notion of distance from thatcolor could be introduced such that pixels with colors close to thebacking color, for example those belonging to the bluescreen that havethe exact backing color or a color close to it, or those where thebluescreen is seen through a transparent object, are considered to bepart of the background for compositing purposes. Pixels with colorsfurther away from the backing color, i.e. those making up the talent,are considered to belong to the foreground. However, it is in practiceimpossible to have a backing color of pure blue and it is usually anarbitrary color close to blue. In order to use such a notion ofdistance, therefore, it would have to be defined between any twoarbitrary points in the RGB cube. Such a distance would becomputationally intensive.

[0167] It is possible however to define a distance between any arbitrarypoint and a specific point, such as the point (0, 0, 1) that is pureblue with maximum saturation and luminance. Equation 1501 shown in FIG.15 defines such a distance value ρ for any pixel C, obtained bysubtracting half the sum of the red and green values from the bluevalue. The higher the distance value, the closer the color is to pureblue. However, since the backing color is never blue, using thisdistance causes problems. For example, point 1106 at (0.2, 0.3, 0.8), atypical backing color, is perceptually an obvious blue. However, it hasthe same distance value (0.55) as the point (0.7, 0.2, 1.0) which ismagenta. Magenta colors often result from reflections of the bluescreenonto blonde hair or fair skin and so the color of a pixel belonging tothe talent could be at the same distance from blue as the backing color.Using ρ is clearly, therefore, not appropriate for chroma-keying in itscurrent form.

[0168] However, if all the pixels in the source frame are hue-shifted asdescribed with reference to FIG. 12, such that the backing color is asblue as possible, this problem is resolved. Pixels with the backingcolor will then be the ones in the image that are closest to blue.Pixels where the backing color is seen through a transparent object willbe relatively close to blue, while the talent, which should contain noneof the backing color, will be a long way from blue.

[0169] In the example, the backing color with co-ordinates at point 1106can be hue-shifted to the point (0.25, 0.25, 0.8), which has a distancevalue of 0.56, by rotating it about line 1105 by 90. Applying the sametransformation to the magenta color gives the point (0.75, 0.2, 0.95).This point has a distance value of 0.48, which is less than the distancevalue of the transformed backing color. The magenta color is thereforefurther away from blue than the backing color.

[0170] Hence it is possible to hue-shift every pixel in the source frameby rotating them about the line 1105 by the angle necessary to make thebacking color as blue as possible, thus giving, for each source framepixel, a corresponding transformed pixel. ρ can then be used on thehue-shifted pixels to give, for each source frame pixel, a distancevalue measuring how far each corresponding transformed pixel is fromblue. These distances can then be compared with the distance of thebacking color from blue to determine which pixels belong to theforeground and which to the background, thus producing a matte.Depending on the distance value of the corresponding transformed pixel,a certain proportion of the backing color can then be subtracted fromeach source frame pixel to color-suppress the source frame 901.

[0171] This technique not only works when the backing color is not auniform blue, but also in those cases when backing colors of differenthues are used. For example, when filming a science fiction movie adirector may want to shoot a blue-skinned alien against a green backingcolor. Transforming the co-ordinate system will move the green backingcolor to blue and the blue skin to red. Thus, the backing color will beconsidered as close to blue and the talent as a long way from blue.

[0172]FIG. 16

[0173] Graph 1601 illustrates how to obtain γ, the angle through whichany color point must be rotated in order to make the color as blue aspossible. Calculating γ for a selected pixel in the background thereforegives the necessary hue-shift angle for the source frame. Referring backto FIG. 12, it is clear that in order to hue-shift a color withoutchanging its saturation or luminance it must be rotated about line 1105.When rotated, a color has a maximum distance value when it appears to besuperimposed on the blue axis when the RGB cube is viewed from the whitecorner. As described with reference to FIG. 13, the rotation isperformed by moving line 1105 onto the blue axis and then rotating thepoint around this line, before returning the co-ordinate system to itsoriginal position.

[0174] Graph 1601 shows the RGB co-ordinate system with the blue axispointing out of the page towards the viewer. Rotation around this axisis now a two-dimensional problem. Point 1602 is point 1106 having beenmoved by steps 1301 and 1302. Angle χ is obtained by the inverse tanfunction of the value on the green axis divided by the value on the redaxis, as shown by equation 1603, where C contains the originalco-ordinates of point 1106 as in FIG. 15. λ, the total measurement fromthe red axis, is obtained as shown at 1604. χ is added to either 2π or πor left unchanged, according to the quadrant in which the transformedpoint lies. This compensates for the periodic nature of the tanfunction.

[0175] Line 1605 illustrates the position of pure blue, which is at 2400from the red axis. γ is therefore obtained by subtracting angle λ from240° as shown by equation 1606. Note that equations 1604 and 1606 areexpressed in terms of radians although in this description degrees areused to facilitate understanding. In the current example, using thepoint (0.2, 0.3, 0.8) as the color of the selected background pixel, γis calculated as 9°.

[0176]FIG. 17

[0177]FIG. 17 again shows the RGB cube 1101 and also shows point 1202,which is point 1106 rotated by 9° about line 1105, ie the pointrepresenting the hue-shifted backing color.

[0178] Comparing point 1202 with point 1106 as shown in FIG. 11, it canbe seen that the point has clearly moved closer to blue. It will also benoted that the red and green components of point 1202 are equal.Referring to FIG. 12, it will be seen intuitively that in order for apoint to appear superimposed on the blue axis when the RGB cube isviewed from the white corner, the red and green components must beequal. Mathematically, it can be shown that when a color is rotatedaround line 1105 by such an angle as to maximise its distance value, thered and green components will always be equal.

[0179]FIG. 18

[0180] A pixel belonging to source frame 901 is defined by a columnmatrix C_(S) with four rows, the first three representing the color'scomponents in the RGB cube and the fourth currently zero. Thehue-shifted source frame pixel, C_(S)′, has been transformed in theabove manner by an angle γ calculated with respect to the color of aselected background pixel.

[0181] The above-described transformation of the source frame pixels isonly necessary to calculate the distance values of the correspondingtransformed pixels. For each source frame pixel a transparency value isthen calculated from the distance value, which is multiplied by thecomponents of the backing color and subtracted from the components ofthe source frame pixel. This last calculation is in terms of theoriginal pixels, not their hue-shifted counterparts, and they wouldtherefore need to be transformed back to their original positions.

[0182] Equation 1801 therefore defines, for a source frame pixel C_(S),a shifted distance value δ. This is the distance value of thecorresponding transformed pixel C_(S)′. Half the red and greencomponents of the shifted color are subtracted from the blue componentof the shifted color. For any pixel in the source frame, therefore, theshifted distance value δ is equal to the distance value ρ of thecorresponding transformed pixel. Using the identity C_(S)′=TC_(S), itcan be shown that δ is defined as in equation 1802, as follows.

[0183] Cos of γ is added to the square root of 3 multiplied by sin of γ.This is halved, negatived and multiplied by the original red componentsof the pixel. Cos of γ is then subtracted from the square root of 3multiplied by sin of γ. This is then halved and multiplied by theoriginal green component of the pixel. The original blue component ofthe pixel is multiplied by cos of γ. These three products are thensummed. Thus the shifted distance value δ of a pixel describes how closeits hue-shifted point C_(S)′ is to blue, but is calculated with respectto its original values C_(S). Note that δ varies only with the color ofthe pixel. γ, the amount by which the source frame pixels arehue-shifted, is obtained from the selected background pixel and is thesame for every pixel in the source frame.

[0184]FIGS. 19A and 19B

[0185]FIGS. 19A and 19B illustrate the idea of the ρ distance within theRGB cube. FIG. 19A illustrates RGB cube 1101 containing plane 1901.Plane 1901 is defined by all points having a distance value of zero. Ascan be seen, ρ is not a simple measurement in space, since, for example,using Euclidean geometry the point (0, 0, 0) is at a distance of onefrom blue, whereas the point (1, 1, 1) is at a distance of root 2 fromblue, but both have the same distance value of zero. The ρ distance is amore intuitive distance, since black, white, all greys in between, thefuschia color at (1, 0, ½) and the aqua color at (0, 1, ½), which areall on plane 1901, are at a δ distance of one from blue.

[0186]FIG. 19B illustrates the two planes 1902 and 1903, which aredefined by all colors having distance values of one half and minus onehalf respectively. It can be seen that according to the ρ distance,cyan, magenta and the saturated blue with low luminance at (0, 0, ½) areall at a distance of one half from blue. Red, green and themediumly-saturated yellow with high luminance at (1, 1, ½) are all at adistance of one and a half from blue.

[0187] Therefore, using the ρ distance, cyan and magenta are relativelyclose to blue, grey, bright pink and greeny-blue are further away andred, green and yellow are furthest away.

[0188] The RGB cube can therefore be thought of as being composed of aninfinite number of planes, each being defined by a different distancevalue. By picking any two of these planes a color-space polygon can bedefined using the edges of the RGB cube. Traditionally, definingpolygons in color-space is a complicated procedure, requiring manyvertices to be defined. The method herein described allows a polygon tobe defined by simply identifying two values of ρ.

[0189] It will be clear to anyone skilled in the art that if requiredthe function defining the ρ distance can be altered to use red or greenas the color to be measured from, instead of blue. Additionally,different functions could be developed allowing, for example, secondarycolors to be used.

[0190]FIG. 20

[0191] Distance values, may therefore be used to color-suppress sourceframe 901. This means removing the backing color and making it blacksuch that when the color-suppressed frame is composited with areplacement background frame 951 the background shows through where thebacking color 117 was before. However, a transparency or opacity maskmust be defined to decide firstly which parts of the source frame 901should be color-suppressed and secondly to decide, in the finalcomposited image, for any pixel position whether the color values shouldbe taken from the color-suppressed source frame or from the replacementbackground frame 951.

[0192] A transparency value, which is equal to the opacity value αsubtracted from one, basically measures how far it is from the color ofa source frame pixel to the color of a selected background pixel butobtaining such values is computationally expensive. However, if all thepixels are hue-shifted this measurement remains the same but can beevaluated in terms of the difference between the shifted distance valueof the background pixel and the shifted distance value of a source framepixel. For any source frame pixel, therefore, a transparency value canbe calculated that is a function of its shifted distance value, ie thedistance value of the corresponding transformed pixel.

[0193] For any source frame pixel, a very simple transparency valuecould be set to one if the shifted distance value of the pixel equalledor exceeded the shifted distance value of the selected background pixeland to zero if not. However, this would only work with source frameshaving very well-defined foreground images. In practice a softnessregion is required where the foreground and replacement backgroundimages are merged slightly to create a natural-looking join. To achievethis transparency values must be between zero and one. A transparencyvalue θ, is therefore defined as shown by equation 2001.

[0194] Firstly an upper threshold τ is defined as the shifted distancevalue of the backing color, δ_(K). A softness value α is defined betweenzero and one, and a lower threshold for θ is then defined as τmultiplied by one minus σ, as shown by equation 2002. For any sourceframe pixel, θ is then defined as the lower threshold subtracted fromthe shifted distance value of that pixel, all divided by the lowerthreshold subtracted from the upper threshold, which simplifies to σmultiplied by τ. The value of θ is clamped such that values below zeroare set to zero, whereas values above one are set to one. This functionis shown at 2003.

[0195] θ is a transparency value for each pixel and is used to obtainthe color-suppressed red, green and blue values for each pixel. If thetransparency value of a particular pixel is zero, the pixel is opaque.Its color-suppressed color will be the same as its original color, andin the final compositing that color will be shown at that position. Ifthe transparency value of pixel is one then it is transparent. Ittherefore belongs to the backing color, and its color is fullysuppressed to black, in order to allow the replacement background toshow through in the final composited image.

[0196] Pixels which have transparency values of between zero and one arein the softness region. Source frame pixels in this region have acertain amount, dependent on θ, of the backing color subtracted fromtheir color to give the color-suppressed pixel color. In the finalcompositing the same amount of the color of the replacement backgroundpixel in the same position is added in.

[0197] Thus, when the picture is composited, pixels with a transparencyvalue of zero will show only the color-suppressed foreground, pictureswith a transparency value of one will show only the replacementbackground image, while pixels in the softness region will contain ablend of both. This has the effect of softening the line around thetalent in addition to allowing semi-transparent objects to be compositedcorrectly

[0198] The softness value σ is adjusted to increase or decrease the sizeof the softness region. If a is equal to zero, there is no softnessregion at all and source frame pixels are either unaltered or the wholeof the backing color is subtracted. If σ is equal to one, the softnessregion is as large as possible. This means that all pixels havingshifted distance values between zero and the upper threshold τ will beconsidered as being in the softness region. Pixels with negative valuesof δ will always be left unaltered in the destination foreground, nomatter how large σ is made. This is because the lower threshold for θwill always be positive since σ and Tare positive.

[0199] Graph 2004 illustrates how the transparency value is changed whenσ is altered. The softness region is indicated by the distance betweenthe lower threshold and the upper threshold, and graph 2004 shows that alarger value of σ increases the size of the softness region.

[0200]FIG. 21

[0201]FIG. 21 shows the threshold and softness planes used tocolor-suppress the the analyst shown in FIG. 1. Since the backing coloris (0.2, 0.3, 0.8) the upper threshold τ is 0.56. Plane 2101 is definedby all points in the RGB cube having a distance value of 0.56.

[0202] The user has set the σ value to be 0.6, and this sets the lowerthreshold for θ to 0.22. Plane 2102 is defined by all points in the RGBcube having a distance value of 0.22.

[0203]FIG. 22

[0204]FIG. 22 illustrates how planes 2101 and 2102 define a polygon2201, using the faces of the RGB cube to form the remaining sides. Thuspixels with a shifted distance value of above 0.56 are completelycolor-suppressed, pixels with a shifted distance value of below 0.22 areunaltered and pixels within the polygon 2201 are in the softness region.The transparency value increases linearly from the back face formed byplane 2102 to the front face formed by plane 2101.

[0205]FIG. 23

[0206] The equation for θ can be rewritten as equation 2301, defined asthe shifted distance value σ divided by both δ and τ, all added to 1minus the reciprocal of σ. Since this is in terms of δ, equation 1802,defining δ, can be substituted in as shown at equation 2302. A matrixC_(θ) can thus be defined for each pixel in a source frame. It containsthe pixel's original RGB components from the matrix C_(S), along with aθ value giving the transparency of the pixel. It is obtained bymultiplying the original pixel values by matrix Me, which is a 4×4matrix containing the identity matrix in the first three rows and thedefinition of θ, as given by equation 2302, in the fourth row. Variablesx, y and z are used to simplify the equation, where x is equal to cos ofγ plus root 3 multiplied by sin of γ, y is equal to cos of γ minus root3 multiplied by sin of γ, and z is equal to cos of γ. Thus, Co=M_(θ)C asshown at 2303.

[0207]FIG. 24

[0208] The well known compositing equation is given at 2401. This statesthat for any pixel in a composited image, its color is given by thecolor of the color-suppressed source frame pixel in that position plusthe color of the replacement background frame pixel in that positionmultiplied by 1 minus the opacity value of the source frame pixel. Thecolor-suppressed source frame is that having the foreground image, inthis case analyst 116, against a black background instead of theoriginal background 117.

[0209] In this case, since the analyst 116 is shot against a nearlyuniform backing color 117, it is a reasonable approximation to say thatmatrix C_(θ) is the result of the color-suppressed source frame beingcomposited with a replacement background image comprising only theuniform backing color. Also, 1 minus the opacity value of the sourceframe pixel is the transparency value. Thus, as shown at equation 2402,C_(θ) is allowed to be equal to the color of the source frame pixel inthat position plus the transparency multiplied by the backing color.

[0210] Rearranging this gives equation 2403, which defines thecolor-suppressed source frame pixel in a particular position as thetransparency value of the pixel θ multiplied by the backing color CK,all subtracted from C_(θ). Since C_(θ), θ and CK are all known, thecolor-suppressed source frame C_(D) can be obtained. This can then beinput into equation 2401, along with an additional frame as areplacement background, to give a final composited pixel value for everypixel in the image.

[0211] Hence each pixel is color-suppressed by subtracting itstransparency value θ multiplied by the backing color from its originalcolors. This is shown by equation 2404, which can be written as a columnmatrix having the following entries: θ multiplied by the red componentof the backing color subtracted from the red component of the sourceframe pixel, θ multiplied by the green component of the backing colorsubtracted from the green component of the source frame pixel, θmultiplied by the blue component of the backing color subtracted fromthe blue component of the source frame pixel, and θ. This is shown byequation 2405.

[0212] The matrix C_(Dθ) can therefore be defined as the product ofmatrices M_(K) ^(A) and C_(θ) added to matrix M_(K) ^(B), as shown byequation 2405. Matrix M_(K) ^(A) is a 4×4 matrix having as its firstthree columns the identity matrix and whose fourth column is −C_(K).Matrix M_(K) ^(B) is a column vector whose first three rows are zero andfourth is one.

[0213]FIG. 25

[0214] Suppressing the backing color of a frame is the most crucial stepin chroma-keying. However when filming against a bluescreen, blue lightoften reflects onto light portions of talent, particularly the skin andblonde hair. This is normally removed by a process known asflare-suppression. Parts of the talent in the foreground image have someof the blue suppressed to produce a flare-suppressed source frame.

[0215] Like chroma-keying, traditional flare-suppression algorithms arecomplicated and cannot be implemented using 4×4 matrices. Typically theprocess involves calculating, for each component in the backing color, acompensating value which should be added on in order to send the backingcolor to a grey of equal luminance. Calculating these compensatingvalues is usually difficult. Again the backing color C_(K) ishue-shifted to C_(K) in order to make the computation easier. Thetransformed backing color has the same luminance as the original backingcolor and therefore removing the saturation from either results inexactly the same shade of grey.

[0216] Greys occur when all three components are equal, and so equation2501 stipulates that C_(Kr) plus a first amount Δ_(r) must equal C_(Kg)plus a second amount Δ_(g) which must equal C_(Kb) plus a third amountΔ_(b). Equation 2502, which stipulates that the sum of the three amountsmust be equal to zero, ensures that the resulting grey has the sameluminance as the backing color. Clearly, therefore, at least one ofΔ_(r), Δ_(g) or Δ_(b) must be negative.

[0217] If these equations can be solved and a mask produced that showsonly the parts of the talent that needs flare-suppressing thenflare-suppression is then a relatively simple matter, but the twoequations have three variables to solve for which normally makes thesolution difficult. However, the result of the hue-shift, as discussedwith reference to FIG. 17, is to make the red and green components ofthe transformed backing color equal. Thus, as shown at equation 2503,Δ_(r) is equal to Δ_(g). There are now three equations to solve thethree variables, and this is a simple task resulting in the solutionsshown at 2504, which are that the first and second amounts, Δ_(r) andΔ_(g), are equal to a third of the threshold value τ, while the thirdamount Δ_(b) is equal to minus two thirds of the threshold value τ.These simple variables can be easily implemented using 4×4 matrices.

[0218]FIG. 26

[0219] The flare-suppression however should not be applied to the entireimage. Only parts of the talent which are bluer than they should be areto be flare-suppressed. Therefore, a mask must be defined, similar tothe color-suppression mask defined by the transparency values θ. Such amask is defined by the flare value β in FIG. 26. The user specifies aflare-suppression value, φ, in addition to the softness value σ. Theupper threshold for β is τ, as with θ, and the lower threshold is φsubtracted from τ multiplied by 1 minus φ multiplied by 1 minus σ. Thesethresholds are shown by equations 2601 and 2602.

[0220] For any pixel, its flare value is defined by the lower thresholdsubtracted from its shifted distance value δ, all divided by the thelower threshold subtracted from the upper threshold, with the resultclamped between zero and one as shown by equation 2603. Graph 2604illustrates the flare-suppression region where δ values are between thelower threshold and the upper threshold. It illustrates how the greaterthe value of φ, the larger the flare-suppression region. Note that,unlike the softness region, the flare-suppression region can extend intonegative values of δ. The definition of the lower threshold for theflare values ensures that it will always be less than (or equal to, ifthe flare-suppression value φ is set to zero) the lower threshold forthe transparency values.

[0221]FIG. 27

[0222] The lower threshold for 3 defines another plane in the RGB cube.FIG. 27 shows, in the current example, plane 2101 given by upperthreshold τ and plane 2102 given by the lower threshold for θ, both asshown in FIG. 21, and also shows plane 2701 which is that given by thelower threshold for β. The user has set a φ value of 0.4, which yields alower bound for β of −0.27, and pixels with δ values of −0.27 are onplane 2701.

[0223]FIG. 28

[0224] The polygon created by the three planes 2101, 2102 and 2701 isshown in FIG. 28. For all pixels having hue-shifted colors outside thepolygon on the blue side, θ and β are both equal to one. These arepixels that belong to the backing color. Pixels with hue-shifted colorsfalling in the first area of the polygon 2201, that defined betweenplane 2101 and plane 2102, have both transparency and flare valuesbetween zero and one. These pixels fall within both the softness regionand the flare-suppression region. Pixels whose hue-shifted colors fallwithin polygon 2802, defined between plane 2101 and plane 2701, havezero transparency values but flare values between zero and one. Thesepixels are not in the softness region but are in the flare-suppressionregion. Transformed pixels with colors outside the polygon on thegreen/red side of the cube have both transparency and flare values setto zero. These pixels belong to areas of the analyst which neitherbelong to the softness region nor should be flare-suppressed.

[0225]FIG. 29

[0226] As for θ, β is defined in terms of the shifted distance value δas shown by equation 2901. β can be written as the lower bound of β,L_(β), divided by L_(β) subtracted from τ, all subtracted from δ dividedby L_(β) subtracted from τ. Matrix C_(β), which contains a pixel'ssource red, green and blue component values plus a flare value β, isdefined as shown by equation 2902. This is a 4×4 matrix that has as itsfirst three rows of the identity matrix and as its last row equation2901 combined with equation 1802 that defines δ. This matrix is calledM_(β), and so C_(β) is defined as M_(β) multiplied by C_(S).

[0227]FIG. 30

[0228] For each source frame pixel color component, the amounts Δ_(r),Δ_(g) and Δ_(b) must be multiplied by its flare value before being addedon to its red, green and blue hue-shifted components respectively. Thus,for example, to obtain the flare-suppressed red component of a pixel itsflare value is multiplied by Δ_(r) and the result is added on to itshue-shifted red component C_(Sr). This is shown by equation 3001, wherea matrix M_(Δ) is multiplied by a matrix C_(β). Matrix M_(Δ) has as itsfirst three columns the 4×4 identity matrix and in its last column ithas the amounts that must be added on to each of the components forflare-suppression, ie Δ_(r), Δ_(g) and Δ_(b). C_(β) contains thehue-shifted color components of the pixel together with the pixel'sflare value β. This gives a matrix C_(φβ), which contains theflare-suppressed pixel color components and the pixel's β value.

[0229] However, these amounts are the amounts that are added onto thetransformed backing color, and the actual compensating values should bethose necessary to send the original backing color to grey. In order toobtain these it is recalled that C_(S) equals TC_(S) as shown byequation 3002. Each side of equation 3001 is then multiplied by theinverse of T, as shown by equation 3003. Since T¹T is equal to one, thisgives the definition of C_(φβ), which is in terms of the original pixelcomponents, as C_(β) multiplied by matrix R multiplied by matrix M_(Δ)multiplied by matrix T¹, as shown by equation 3004.

[0230] It can be shown that the matrix T¹M_(Δ)T can be written as matrixM_(φ) as shown by equation 3005. Matrix M_(φ) has as its first threecolumns the 4×4 identity matrix, while its fourth column contains thethree compensating values and one. The first compensating value is Δ_(r)multiplied by x, the second compensating value is Δ_(g) multiplied by y,and the third compensating value is Δ_(b) multiplied by z, where x, yand z are as defined in FIG. 23.

[0231]FIG. 31

[0232] For any pixel in source frame 901, matrix C_(φβ) contains theflare-suppressed red, green and blue values plus the flare value β.However, once flare suppression has taken place flare values are nolonger necessary and transparency values are needed in order to carryout the color suppression. Equation 3101 defines the transparency valueθ in terms of β, which is obtained by eliminating δ. θ is equal to theupper threshold τ minus the lower threshold for β divided by both σ andτ, subtracted from one, and all added to β multiplied by the upperthreshold τ minus the lower threshold of β divided by both σ and τ. Thisis substituted into equation 3005 to give equation 3102, which definesmatrix C_(φβ). This matrix contains, for any pixel, its flare-suppressedred, green and blue components along with its transparency value. It isobtained by multiplying matrix C_(β) by a matrix M_(φ) ^(A) and addingthe result to matrix M_(φ) ^(B). Matrix M_(φ) ^(A) is identical tomatrix M_(φ) except that the bottom right value is not one but the upperthreshold τ minus the lower threshold of β divided by both σ and τ.Matrix M_(φ) ^(B) is a column vector whose first three values are zeroand last value is the upper threshold τ minus the lower threshold of βdivided by both σ and τ, all subtracted from one. Thus, at this stage,the matrix C_(φβ) contains the already flare-suppressed color componentsand a θ value that can be used to perform color suppression.

[0233]FIG. 32

[0234] Referring back now to FIG. 24, the matrix transformations forcolor suppression are given by equation 2406. If flare suppression hasalready taken place then a proportion of the backing color will havealready been subtracted from pixels that are to be color-suppressed,since the color-suppression region falls completely within theflare-suppression region. This proportion is equal, for each component,to the flare value multiplied by the compensating value for thatcomponent. Replacing C_(θ) with C_(φθ) (which contains theflare-suppressed pixel color values) and compositing matrix M_(K) ^(A)with matrix M_(φ) ^(A) to give matrix M_(K) ^(C), as shown in equation3201, color-suppresses, according to the mask defined by thetransparency values, the already flare-suppressed source frame.

[0235]FIG. 33

[0236] Graph 3301 plots transparency values θ and flare values β againstshifted distance values δ. For values between −1 and L_(β), the lowerthreshold for β, both θ and β are zero. Pixels with shifted distancevalues falling into this region are neither flare-suppressed norcolor-suppressed and therefore belong to areas of the analyst that donot need correcting.

[0237] Pixels with shifted distance values between the lower bound forβ, L_(β), and the lower bound for θ, L_(θ), have zero transparencyvalues but flare values between zero and one. These pixels are notcolor-suppressed but are flare-suppressed and therefore belong to areasof the analyst that have blue reflections but are not in the softnessregion.

[0238] Pixels with shifted distance values between L_(θ) and the upperthreshold τ have both transparency and flare values between zero andone. These pixels fall into the softness region on the edge of theanalyst and also require flare suppression. Pixels with shifted distancevalues between the upper threshold τ and 1 have both transparency andflare values of one. These pixels are fully color-suppressed andflare-suppressed and in the fully suppressed source frame will be blackor very close to black. These are pixels that have the backing color.

[0239]FIG. 34

[0240]FIG. 34 details the effect of the color-suppression andflare-suppression in terms of the final image, which is a blending ofthe fully suppressed (ie flare-suppressed and color-suppressed) sourceframe 901 with a certain amount of the replacement background frame 951,determined by the transparency values θ. Equation 3401 gives, as anexample, the final value of the red component of a pixel after beingflare-suppressed, color-suppressed and composited with a pixel from thereplacement background frame in the same position. For any pixel, itsfinal red component C_(Fr) is equal to the fully suppressed source framered component C_(Dr) added to the replacement background frame redcomponent C_(Br) multiplied by, 1 minus the opacity α. The fullysuppressed source frame red component C_(Dr) is obtained from equation3201 with equation 3102 substituted in, and is θ multiplied by the redcomponent of the backing color C_(Kr) subtracted from the red componentof the source frame pixel C_(Sr), all added to a third of τ multipliedby x (the first compensating value) multiplied by θ subtracted from β.Substituting this into the compositing equation and replacing 1 minus αby θ gives equation 3401.

[0241] A pixel with both θ and β equal to zero belongs to the foregroundimage, ie analyst 116, and does not require any sort of colorcorrection. Therefore the final composited red component C_(Dr) simplyequals the original source frame red component C_(Sr). Since the opacityof this pixel is one none of the background image is added, and sinceits flare value is zero it is not flare-suppressed. This is shown atequation 3402.

[0242] If a pixel has a transparency value of zero but a flare valuebetween zero and one it belongs to a part of the analyst 116 thatcontains blue spill but does not belong to the softness region. Theopacity of a pixel in this area is one and so the background image doesnot show through. As shown by equation 3403, the fully suppressed redcomponent C_(Dr) is therefore equal to the source red component C_(Sr)added to a third of β multiplied by τ multiplied by x. It will berecalled that a third of τ multiplied by x is the first compensatingvalue that must be added on to the red component of the backing colorduring the process of sending the backing color to grey. In theflare-suppressed regions a proportion, given by β, of this amount isadded to the red component. Since β increases the more blue the pixelis, this controls how much flare-suppression is applied to each pixel.

[0243] Pixels that have both β and θ values between zero and one belongto the softness region of analyst 116. The final composited redcomponent C_(Dr) in this region is that given by equation 3401. Sincethe pixel is in the softness region the final color should be a mixtureof the source frame pixel color and the replacement frame pixel color,but it should also be flare-suppressed. Therefore a smaller proportionof the first compensating value is added than for pixels not in thesoftness region, the proportion being given by θ subtracted from β. Aproportion, given by θ, of the backing color red component C_(Kr) issubtracted and the same amount of the red component C_(Br) of thereplacement background pixel at that position is then added on.

[0244] Pixels that have both β and θ equal to one belong to thebackground 117 and have the backing color or a color very close to it.Here the compensating value is cancelled out and so the fully suppressedred component C_(Dr) is given by the backing color red component C_(Kr)subtracted from the source frame red component C_(Sr), the result ofwhich is approximately equal to zero. Similarly, the fully suppressedgreen and blue components are approximately equal to zero. The fullysuppressed color C_(D) for these pixels is therefore black. Since thesepixels have an opacity of zero the background image shows throughcompletely. In the final image, therefore, the colors of these pixelsare equal or very close to the colors of the corresponding pixels of thereplacement background frame 951.

[0245]FIG. 35

[0246]FIG. 35 shows the three calculations that must be performed tofully suppress (ie color-suppress and flare-suppress) a source frame.Firstly, as shown by equation 3501, the source foreground pixel C_(S) ismultiplied by matrix M_(β) (defined in FIG. 29). The values are thenclamped between zero and one. This gives the flare-suppression maskprovided by the flare values β. The color components of the pixel areunchanged at this point.

[0247] Secondly, as shown by equation 3502, the result of 3501 ismultiplied by the matrix M_(φ) ^(A) and added to matrix M_(φ) ^(B) (bothdefined in FIG. 31). The values are then clamped between zero and one.This gives the flare-suppressed red, green and blue components asdetermined by the flare-suppression mask, and also gives thecolor-suppression mask provided by the transparency values θ. At thispoint, therefore, the output is the flare-suppressed pixels of thesource frame and the transparency mask.

[0248] Thirdly, as shown by equation 3503, the result of 3502 ismultiplied by matrix M_(K) ^(A) and added to matrix M_(K) ^(B) (bothdefined in FIG. 24). The values are then clamped between zero and one.This color-suppresses the red, green and blue components according tothe color-suppression mask, and also gives an opacity value α, which isequal to 1 minus θ. The output at this stage is the fully suppressedsource frame with a matte provided by the opacity values.

[0249] The opacity value is then used to blend the fully suppressedsource frame, comprising only the foreground of analyst 116 over a blackbackground, with the replacement background frame 951, as described inFIG. 34. The transparency mask given by θ can be kept instead of theopacity mask α but most compositing software uses opacity values.Alternatively, the fully suppressed source frame matrix C_(D) can besplit into two matrices, one containing only the color values and theother containing only the opacity value. These matrices can then be usedto blend the foreground and background as described with reference toFIG. 9.

[0250]FIG. 36

[0251] It is possible to perform the color-suppression and theflare-suppression separately. Firstly, letting φ be equal to zero makesall the pixels' β and θ values equal. Referring back to FIG. 34 it willbe seen that this eliminates the flare-suppression. However, the twoprocesses may also be completely separated. FIG. 36 shows thecalculations necessary to perform only color suppression. Firstly, asshown by equation 3601, the source foreground pixel C_(S) is multipliedby matrix M_(θ) (defined in FIG. 23). The values are then clampedbetween zero and one. This gives the color-suppression mask provided bythe transparency values but at this stage the color components of thepixel are unchanged.

[0252] Secondly, as shown by equation 3602, the result of equation 3601is multiplied by matrix M_(K) ^(A) and added to matrix M_(K) ^(B) (bothdefined in FIG. 24). The values are then clamped between zero and one.This color-suppresses the source foreground according to thecolor-suppression mask and also gives an opacity value α. The output istherefore only color-suppressed and not flare-suppressed and alsocontains a matte given by the opacity values.

[0253]FIG. 37

[0254]FIG. 37 shows the calculations necessary to perform only flaresuppression. Firstly, as shown by equation 3701, the source frame pixelC_(S) is multiplied by matrix M_(β) (defined in FIG. 29). The values arethen clamped between zero and one. This gives the flare-suppression maskprovided by the flare values. The color components of the pixel areunchanged at this point.

[0255] Secondly, as shown by equation 3702, the result of equation 3701is multiplied by matrix M_(φ) ^(C) and added to matrix M_(φ) ^(D).Matrix M_(φ) ^(C) is identical to matrix M_(φ) ^(A) (defined in FIG. 31)except that the fourth value of the fourth column is negative, as shownby identity 3703. Matrix M_(φ) ^(D) is identical to matrix M_(φ) ^(B)(defined in FIG. 31) except that the fourth value is τ minus the lowerthreshold for β, all divided by both σ and τ, as shown by identity 3704.The results are then clamped between zero and one. This flare-suppressesthe source frame according to the flare-suppression mask, and alsoprovides an opacity value α.

[0256] The flare-suppressed frame output by this process will have anear-uniform grey background. It cannot be immediately composited with abackground image using the compositing equation given here but theoutput includes opacity values and so the frame may be used with anothercompositing technique.

[0257]FIG. 38

[0258] Returning to the implementation of the compositing on a graphicsaccelerator, the calculations shown in FIGS. 35 to 37 are converted intofunctions and are implemented in fragment processor 310. However firstthe user must interact with the application. Referring back to FIG. 4,the user chooses the keying method at step 405. He can choose betweendifference keying, luminance keying, full chroma-keying, colorsuppression only or flare suppression only. Difference keying andluminance keying have already been described with reference to FIG. 8.For any of the types of chroma-keying, the parameters are initialised asfollows. The backing color is assumed to be pure blue at (0, 0, 1) andthe threshold value τ is therefore set at one. The softness value σand/or the flare-suppression value φ are both set to zero, depending onthe type of suppression chosen.

[0259] A source frame 901 is received at step 406 and at step 407 thekeying application is interacted with. This is detailed in FIG. 38. Atstep 3801 a question is asked as to whether the user has changed any ofthe parameters. If this question is answered in the negative thencontrol is directed to step 408. However, if it is answered in theaffirmative then at step 3802 a question is asked as to whether the userhas changed the backing color. The user may do this either by selectinga pixel in the source foreground image or by altering the RGB values asshown in FIG. 7. It is also contemplated that the backing color may beautomatically selected by the application. If this question is answeredin the affirmative the hue-compensation angle γ for that backing coloris calculated at step 3803 and at step 3804 the upper threshold τ iscalculated. τ is re-calculated when the backing color changes but canthen be altered by the user if required.

[0260] At this point, and if the question asked at step 3802 is answeredin the negative, meaning that only the parameters σ, τ or φ have beenchanged, then at step 3805 the variables used in the various matricesare calculated based on the backing color and the parameters γ, σ, τ andφ.

[0261] Control then proceeds to step 408 when the source frame 901 issent as a texture to the GPU along with the variables identified at step3805.

[0262]FIG. 39

[0263] A standard graphics card is used to perform chroma-keying byinputting the source frames as textures and associating them withcertain functions. Steps carried out by a fragment processor are shownin Figure H. For each fragment the appropriate function is identified atstep H1 by reading the fragment header, and the fragment is processed atstep H5. FIGS. 39 to 44 therefore show functions carried out by thefragment processor at step H5.

[0264] When the user has selected full chroma-keying the fragmentprocessor carries out the calculations identified in FIG. 35 byperforming three passes. FIG. 39 shows the function carried out in thefirst of these passes. At step 3901 a source frame pixel C_(S) ismultiplied by matrix M_(β) and at step 3902 the resultant values areclamped between zero and one. Returning to Figure H, the pixel is thendrawn in the frame buffer at step H6 and the fragment is copied to thetexture processor at step H8, since this function calls for a reprocess.Once all pixels have been processed in this way the new texture,comprising the processed fragments, is input back into the fragmentprocessor to be processed as shown in FIG. 40.

[0265]FIG. 40

[0266]FIG. 40 illustrates the second function carried out by thefragment processor. At step 4001 the clamped matrix C_(β) is multipliedby the matrix M_(φ) ^(A). At step 4002 the result of step 4001 is addedto matrix M_(φ) ^(B). At step 4003 the resultant values are clampedbetween zero and one. Once all pixels have been processed in this waythe new texture is input back into the fragment processor to beprocessed as shown in FIG. 41.

[0267]FIG. 41

[0268] At step 4101 the clamped matrix C_(φθ) is multiplied by matrixM_(K) ^(C). At step 4102 the resultant values are added to matrix M_(K)^(B). At step 4103 the resultant values are clamped between zero andone. Once all pixels have been processed in this way they are written tothe frame buffer. In this case the function does not call for areprocess. The fully suppressed source frame is therefore left in theframe buffer to be blended with the replacement background image.Alternatively, it can be split into two layers, one containing the colorinformation and one containing the matte information, before blending.

[0269] In some more advanced fragment processors it is possible toperform all three steps in a single pass. In that case the threefunctions shown in FIG. 35 are combined into a single function whichdoes not call for a reprocess.

[0270]FIG. 42

[0271] If the user has selected only flare suppression the fragmentprocessor carries out the calculations identified in FIG. 36 byperforming two passes. The function carried out at the first pass is thesame as that shown in FIG. 39. FIG. 42 shows the function carried out inthe second of these passes.

[0272] At step 4201 the clamped matrix C_(β) obtained on the first passis multiplied by matrix M_(φ) ^(C). At step 4202 the resultant valuesare added to matrix M_(φ) ^(D). At step 4203 the resultant values areclamped between zero and one. Once all pixels have been processed inthis way they are written to the frame buffer to be processed further insome way.

[0273]FIG. 43

[0274] If the user has selected only color suppression the fragmentprocessor carries out the calculations identified in FIG. 37 byperforming two passes. FIG. 43 shows the function carried out in thefirst of these passes.

[0275] At step 4301 the source frame pixel C_(S) is multiplied by matrixM_(θ). At step 4302 the resultant values are clamped between zero andone. Once all pixels have been processed in this way the new texture isinput back into the fragment processor to be processed as shown in FIG.44.

[0276]FIG. 44

[0277] At step 4401 the clamped matrix C_(θ) is multiplied by matrixM_(K) ^(A). At step 4401 the resultant values are added to matrix M_(K)^(B). At step 4403 the resultant values are clamped between zero andone. Once all pixels have been processed in this way they are written tothe frame buffer, either as a single layer or as two layers, foregroundand matte, to be blended with the background image.

[0278]FIG. 45

[0279] With reference to the description to FIG. 1, an aim of theprocessing application 504 according to the present invention is tofacilitate the generation of a matte from a source frame 901 tocomposite a portion of said frame, defined as a foreground image, with areplacement background 951. A typical application for which thecompositing function of the present invention is provided, is shown inFIG. 45, including a dynamic internet page configured with a webcastingportion displayed within an internet browser.

[0280] A VDU 4501 is shown and displays the GUI 4502 of a computer'soperating system, comparable in purposes and functionality to operatingsystem 501. Said GUI 4502 includes a plurality of user-actionablewidgets 4503, representing standard operating system tasks. A second GUI804 of an internet browsers comparable in purposes and functionality tobrowser 502 is shown overlaid on GUI 4502 and features a plurality ofconventional browser tasks widgets, including a “go to my home page”widget 4505, a “stop navigation” widget 4506 and navigation widgets 807.In the example, the browser 4504 displays an active server page (ASP)4508 broadcast over the internet 108 by the financial website describedin FIG. 1.

[0281] Accordingly, ASP 4508 features financial information in differentformats, which include for instance a bar chart 4509, pie charts 4510and a statistical comparison chart portion 4511. In the example, charts4509, 4510 and statistical data in portion 4511 are updated in real-timefrom various stockmarkets around the world. Analyst 116 shown in FIG. 1is commissioned to comment the financial information described above inreal-time also and is optionally identified by name shown at 4512. In analternative embodiment of the present invention, a plurality of analystsare similarly commissioned around the world to simultaneously providethe same real-time commenting, for instance in their respective naturaltongue, and are thus selectively broadcast over the internet to variousgeographical areas but composited over the same background, i.e. ASP4508. In yet another alternative embodiment the background may be atwo-dimensional representation of a three-dimensional “virtual set”.

[0282] In the figure, analyst 116 is shown partially overlaying barchart 4509. The reason for this partial obstruction is that analyst 116is captured against the blue background 117, wherein in the finalcomposited frame analyst 116 constitutes the foreground image and thebluescreen 117 constitutes the backing color keyed out and replaced withASP 4508 according to the present invention.

1. Image editing apparatus, comprising processing means, input means andselection means; wherein a source frame comprising a foreground imageagainst a background of a substantially uniform backing color isprovided to said processing means via said input means, wherein saidsource frame is composed of a plurality of pixels, each pixel beingrepresented by three components defining a position within a colorspace; wherein said processing means is configured to: (a) calculate atransformation that maximises a distance value of a selected pixel inthe background, which transformation when applied to any pixel in saidsource frame defines a corresponding transformed pixel, wherein for anypixel said distance value describes how close its color is to aspecified color; (b) calculate a transparency value for each pixel inthe source frame, wherein for each said pixel said transparency value isa function of the distance value of the corresponding transformed pixel;and (c) for each pixel in the source frame, multiply the components ofsaid selected background pixel by the transparency value of the sourceframe pixel and subtract the results from the components of the sourceframe pixel to color-suppress said source frame.
 2. Image editingapparatus according to claim 1, wherein said transformation is a hueshift that leaves saturation and luminance unaltered.
 3. Image editingapparatus according to claim 1, wherein said specified color is aspecified primary color, selected from the group of primary colors red,green and blue, with maximum saturation and luminance.
 4. Image editingapparatus according to claim 3, wherein said components are amounts ofsaid three primary colors red, green and blue.
 5. Image editingapparatus according to claim 4, wherein for any pixel in said sourceframe said distance value is defined as the component that representssaid specified primary color minus half of each of the remainingcomponents.
 6. Image editing apparatus according to claim 5, whereinsaid specified primary color is blue.
 7. Image editing apparatusaccording to claim 5, wherein for each pixel in said source frame saidtransparency value is defined as a lower threshold subtracted from thedistance value of the corresponding transformed pixel, all divided bysaid lower threshold subtracted from an upper threshold, with the resultclamped between zero and one.
 8. Image editing apparatus according toclaim 7, wherein said lower threshold is defined as the product of oneminus a constant and said upper threshold.
 9. Image editing apparatusaccording to claim 8, wherein said upper threshold is defined as thedistance value of said transformed selected background pixel.
 10. Imageediting apparatus according to claim 1, wherein said transparency valuesare used to produce a matte.
 11. Image editing apparatus according toclaim 1, wherein an additional frame is provided to said processingmeans via said input means, wherein said additional frame is composed ofa plurality of pixels, each pixel being represented by three componentsdefining a position within the color space and each pixel correspondingin position to a pixel in said source frame; wherein said processingmeans is additionally configured, for each pixel in the color-suppressedsource frame, to: multiply the components of the corresponding pixel insaid additional frame by said transparency value of the source framepixel; and add the results to the color-suppressed components of saidsource frame pixel, such that said source frame is composited with saidadditional frame.
 12. Image editing apparatus according to claim 1,wherein said processing means is a graphics accelerator card.
 13. Imageediting apparatus, comprising processing means, input means andselection means; wherein a source frame comprising a foreground imageagainst a background of a substantially uniform backing color isprovided to said processing means via said input means, wherein saidsource frame is composed of a plurality of pixels, each pixel beingrepresented by three components defining a position within the RGB colorspace; wherein said processing means is configured to: (a) calculate atransformation of a selected pixel in the background that maximises thedistance value of said selected background pixel while leaving itssaturation and luminance unchanged, which transformation when applied toany pixel in said source frame defines a corresponding transformedpixel, wherein: for any pixel said distance value describes how closeits color is to a specified primary color with maximum saturation andluminance, and said transformation is a hue shift that leaves saturationand luminance unaltered; (b) calculate a transparency value for eachpixel in the source frame, wherein for each said pixel said transparencyvalue is a function of the distance value of the correspondingtransformed pixel; and (c) for each pixel in the source frame, multiplythe components of said selected background pixel by the transparencyvalue of the source frame pixel and subtract the results from thecomponents of the source frame pixel to color-suppress said sourceframe.
 14. Image editing apparatus according to claim 14, wherein forany pixel said distance value is defined as the component representingsaid specified primary color minus half of each of the remainingcomponents.
 15. Image editing apparatus according to claim 14, whereinfor any pixel said transparency value is defined as a minimum thresholdsubtracted from the distance value of the corresponding transformedpixel, all divided by said minimum threshold subtracted from a maximumthreshold, with the result clamped between zero and one, wherein: saidminimum threshold is defined as the product of one minus a constant andsaid maximum threshold, and said maximum threshold value is defined asthe distance value of said transformed selected background pixel. 16.Image editing apparatus according to claim 14, wherein an additionalframe is provided to said processing means via said input means, whereinsaid additional frame is composed of a plurality of pixels, each pixelbeing represented by three components defining a position within thecolor space and each pixel corresponding in position to a pixel in saidsource frame; wherein said processing means is additionally configured,for each color-suppressed pixel in the color-suppressed source frame,to: multiply the components of the corresponding pixel in saidadditional frame by said transparency value for the source frame pixel;and add the results to the color-suppressed components of the sourceframe pixel, whereby said source frame is composited with saidadditional frame.
 17. A method of color-suppressing a source framecomprising a foreground image against a background of a substantiallyuniform backing color, wherein said source frame is composed of aplurality of pixels and each pixel is represented by three componentsdefining a position within a color space; comprising the steps of: (a)calculating a transformation that maximises a distance value of aselected pixel in the background, which transformation when applied toany pixel in said source frame defines a corresponding transformedpixel, wherein for any pixel said distance value describes how close itscolor is to a specified color; (b) calculating a transparency value foreach pixel in the source frame, wherein for each said pixel saidtransparency value is a function of the distance value of thecorresponding transformed pixel; and (c) for each pixel in the sourceframe, multiplying the components of said selected background pixel bythe transparency value of the source frame pixel and subtracting theresults from the components of the source frame pixel.
 18. A method ofcolor-suppressing a source frame according to claim 17, wherein saidtransformation is a hue shift that leaves saturation and luminanceunaltered.
 19. A method of color-suppressing a source frame according toclaim 17, wherein said specified color is a specified primary color,selected from the group of primary colors red, green and blue, withmaximum saturation and luminance.
 20. A method of color-suppressing asource frame according to claim 19, wherein said components are amountsof said three primary colors red, green and blue.
 21. A method ofcolor-suppressing a source frame according to claim 20, wherein for anypixel the distance value of said pixel is defined as the component thatrepresents said specified primary color minus half of each of theremaining components.
 22. A method of color-suppressing a source frameaccording to claim 21, wherein said specified primary color is blue. 23.A method of color-suppressing a source frame according to claim 21,wherein said transparency value is defined, for any pixel, as a minimumthreshold subtracted from the distance value of the correspondingtransformed pixel, all divided by said minimum threshold subtracted froma maximum threshold, with the result clamped between zero and one.
 24. Amethod of color-suppressing a source frame according to claim 23,wherein said minimum threshold is defined as the product of one minus aconstant and said maximum threshold.
 25. A method of color-suppressing asource frame according to claim 24, wherein said maximum threshold isdefined as the distance value of the transformed selected backgroundpixel.
 26. A method of color-suppressing a source frame according toclaim 17, comprising the additional step of using said transparencyvalues to produce a matte.
 27. A method of compositing an additionalframe with a source frame that has been color-suppressed according toclaim 17, wherein said additional frame is composed of a plurality ofpixels, each pixel being represented by three components defining aposition within the color space and each pixel corresponding in positionto a pixel in said source frame; in which the following steps areperformed for each pixel in said color-suppressed source frame:multiplying the components of the corresponding pixel in said additionalframe by said transparency value for the source frame pixel; and addingthe results to the color-suppressed components of the source framepixel.
 28. A method of color-suppressing a source frame comprising afront image against a background of a substantially uniform backingcolor, wherein said source frame is composed of a plurality of pixelsand each pixel is represented by three components defining a positionwithin the RGB color space; comprising the steps of: (a) calculating atransformation that maximises a distance value of a selected pixel inthe background, which transformation when applied to any pixel in saidsource frame defines a corresponding transformed pixel, wherein: for anypixel said distance value describes how close its color is to aspecified primary color with maximum saturation and luminance, and saidtransformation is a hue shift that leaves saturation and luminanceunaltered; (b) calculating a transparency value for each pixel in thesource frame, wherein for each said pixel said transparency value is afunction of the distance value of the corresponding transformed pixel;and (c) for each pixel in the source frame, multiplying the componentsof said selected background pixel by the transparency value of thesource frame pixel and subtracting the results from the components ofthe source frame pixel.
 29. A method of color-suppressing a source frameaccording to claim 28, wherein for any pixel said distance value isdefined as the component representing said specified primary color minushalf of each of the remaining components.
 30. A method ofcolor-suppressing a source frame according to claim 28, wherein for anypixel said transparency value is defined as a minimum thresholdsubtracted from the distance value of the corresponding transformedpixel, all divided by said minimum threshold subtracted from a maximumthreshold, with the result clamped between zero and one, wherein: saidminimum threshold is defined as the product of one minus a constant andsaid maximum threshold, and said maximum threshold value is defined asthe distance value of the transformed selected background pixel.
 31. Amethod of compositing an additional frame with a source frame that hasbeen color-suppressed according to claim 28, wherein said additionalframe is composed of a plurality of pixels, each pixel being representedby three components defining a position within the color space and eachpixel corresponding in position to a pixel in said source frame; inwhich the following steps are performed for each pixel in saidcolor-suppressed source frame: multiplying the components of thecorresponding pixel in said additional frame by said transparency valuefor the source frame pixel; and adding the results to thecolor-suppressed components of the source frame pixel, such that saidsource frame is composited with said additional frame.